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This  Note  describes  the  testing  and 
evaluation  of  a  set  of  equations  developed 
for  use  in  the  Air  Force's  Enlisted  Force 
Management  System.  The  equations  predict 
annual  airman  loss  and  reenlistment  rates 
by  Air  Force  Specialty  Code.  Predictions 
from  the  loss  models  are  compared  to  actual 
loss  rates  and  to  loss  rates  estimated  by 
the  model  currently  being  used  by  the  Air 
Force . 


PREFACE 


This  Note  describes  the  testing  and  evaluation  of  a  set  of  equations  that  predict 
airman  loss  and  recnlistmcnt  behavior.  The  equations  were  developed  for  use  in  the  Air 
Force’s  Enlisted  Force  Management  System  (EFMS).  The  conceptual  design  of  the 
EFMS  includes  a  variety  of  loss  models  distinguished  by  the  time  horizon  of  their 
predictions  (short-,  middle-,  or  long-term)  and  whether  such  predictions  are 
disaggregated  by  occupational  specialty. 

This  document  concerns  the  middle-term  disaggregate  loss  model,  which  predicts 
annual  loss  rates  by  Air  Force  Specialty  Code  (AFSC).  Its  predictions  are  designed  to  be 
most  accurate  between  one  and  four  years  into  the  futurc.^For  an  overview  of  the  EFMS, 
sec  Grace  M.  Carter,  Jan  M.  Chaiken,  Michael  P.  Murray,  and  Warren  E.  Walker, 
Conceptual  Design  of  an  Enlisted  Force  Management  System  for  the  Air  Force ,  The 
RAND  Corporation,  N-2005-AF,  August  1983. 

The  test  and  evaluation  results  in  this  Note  do  not  indicate  how  the  models 
will  perform  in  the  operational  EFMS.  The  models  tested  here  arc  those  documented 
in  Grace  Cancr,  Michael  Murray,  R.  Yilmaz  Argiiden,  Marygail  Brauncr,  Harvey 
Greenberg,  Deborah  Skollcr,  Middle-Term  Loss  Prediction  Models  for  the  Air  Force 
Enlisted  Management  System :  Specification  and  Estimation ,  The  RAND  Corporation, 
R-3482-AF,  December  1987.  The  operational  models  will  differ  in  parameter  values, 
structure,  and  performance,  partly  because  of  changes  in  specification  that  resulted  from 
the  outcome  of  the  test  and  evaluation  process.  It  is  also  partly  because  the  models  will 
be  updated  with  more  recent  data  before  their  use  in  the  EFMS.  Subsequent  test  and 
evaluation  results  for  the  revised  models  will  appear  in  future  documents. 

This  Note  should  be  useful  to  three  classes  of  readers: 

1.  Persons  interested  in  the  performance  of  the  set  of  middle-term  disaggregate 
loss  models  that  existed  in  May  1986. 

2.  Analysts  charged  with  improving  those  models. 

3.  Persons  responsible  for  evaluating  future  versions  of  the  models. 


I 


Although  the  Note  contains  a  few  technical  sections,  most  of  it  is  simple  and 
straightforward  and  an  extensive  statistical  background  is  not  required. 

The  work  described  here  is  part  of  the  Enlisted  Force  Management  Project 
(EFMP),  a  joint  effort  of  the  Air  Force  (through  the  Deputy  Chief  of  Staff  for  Personnel) 
and  The  RAND  Corporation.  RAND’s  work  falls  within  the  Resource  Management 
Program  of  Project  AIR  FORCE.  The  EFMP  is  part  of  a  larger  body  of  work  in  that 
program  concerned  with  the  effective  utilization  of  human  resources  in  the  Air  Force. 


SUMMARY 


Air  Force  planners  use  inventory  projection  models  (IPMs)  to  predict  how  an 
initial  inventory  of  airmen  will  look  in  the  future.  Such  IPMs  contain  loss  models  that 
estimate  how  many  members  of  the  initial  inventory  leave  the  service  during  the  period 
in  question.  Middle-term  disaggregate  loss  models  predict  annual  loss  rates,  by  Air 
Force  Specialty  Code  (AFSC)  and  certain  other  factors. 

In  1983,  as  part  of  a  larger  effort  to  develop  a  new  Enlisted  Force  Management 
System  (EFMS)  for  the  Air  Force  (Carter  et  al.,  1983),  RAND  began  developing  a  set  of 
middle-term  disaggregate  loss  models  for  use  in  the  EFMS’s  Middle-Term  Disaggregate 
IPM.  This  document  describes  the  testing  and  evaluationTT&E)  of  these  loss  models. 
The  purpose  of  the  T&E  exercise  was  to  identify  problems  with  the  models  so  that 
the  problems  could  be  fixed  before  the  models  were  implemented  in  the  EFMS. 

The  T&E  showed  that  the  loss  models  as  they  stood  in  May  of  1986  were  good 
enough  to  be  encouraging,  but  not  good  enough  to  be  satisfying.  In  general,  they 
performed  at  least  as  well  as  the  disaggregate  loss  models  used  in  ALPS.  But  the 
exercise  identified  a  number  of  specific  problems,  some  of  which  have  been  corrected 
already,  some  of  which  are  in  the  process  of  being  corrected,  and  some  of  which  will 
receive  attention  in  the  future.  By  the  time  the  EFMS  becomes  operational,  the  models 
evaluated  here  will  have  been  modified  and  updated.  Their  performance  will  differ  from 
that  shown  here. 
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GLOSSARY 


AFSC 

Air  Force  Specialty  Code 

ALPS 

Airman  Loss  Probability  System 

CFG 

Career  Field  Group 

EFMS 

Enlisted  Force  Management  System 

ETS 

Expiration  of  Term  of  Service 

FY 

Fiscal  Year 

IPM 

Inventory  Projection  Model 

PE 

Prediction  Error 

PEPNR 

Prediction  Error  as  Percent  of  Number  at  Risk 

PRE 

Percent  Relative  Error 

SPE 

Standardized  Prediction  Error 

SSN 

Social  Security  Number 

YAR 

Year  at  Risk 

YOS 

Years  of  Service 

I.  INTRODUCTION 


BACKGROUND 

Air  Force  personnel  managers  use  inventory  projection  models  (IPMs)  to  predict 
how  an  initial  inventory  of  airmen  will  look  in  the  future.  Such  IPMs  contain  loss  models 
that  estimate  how  many  members  of  the  initial  inventory  leave  the  service  by  the  future 
period  in  question.  These  loss  models  may  differ  by  time  horizon  and  by  the  extent  to 
which  the  loss  rates  are  aggregated.  Middle-term  disaggregate  loss  models  predict  annual 
loss  rates  by  Air  Force  Specialty  Code  (AFSC)  and  certain  other  factors. 

In  1983,  as  part  of  a  larger  effort  to  develop  a  new  Enlisted  Force  Management 
System  (EFMS)  for  the  Air  Force  (Carter  et  al.,  1983),  RAND  began  developing  a  set  of 
middle-term  disaggregate  loss  models.  Specification  and  estimation  of  these  models 
continues  to  this  day.  This  Note  describes  the  test  and  evaluation1  of  these  models  as 
they  were  in  May  1986.  The  models  tested  are  those  specified  in  Carter  et  al.,  1987. 

The  testing  and  evaluation  described  here  showed  that  the  loss  models  as  they 
stood  in  May  1986  were  good  enough  to  be  encouraging,  but  not  good  enough  to  be 
satisfying.  Few  of  the  models  were  actually  validated,  but  in  general  they  performed  at 
least  as  well  as  the  disaggregate  loss  models  used  in  ALPS.  The  exercise  was  primarily 
diagnostic,  it  identified  several  specific  problems,  some  of  which  have  been  corrected 
already,  some  of  which  are  being  corrected,  and  some  of  which  will  receive  attention  in 
the  future.  These  corrections  should  result  in  models  whose  performance  is  clearly 
superior  to  the  ones  evaluated  here.  By  the  time  the  EFMS  becomes  operational,  the 
models  evaluated  here  will  have  been  modified  and  updated.  Their  performance  will 
differ  from  that  shown  here. 

This  Note  also  describes  how  the  evaluation  was  performed.  Work  on  the  models 
continues,  and  the  Air  Force  intends  to  periodically  rcspccify  and  refit  the  models  as  new 
data  become  available  and  the  nature  of  the  enlisted  force  changes.  This  document 
shows  by  example  how  the  new  models  can  be  tested,  how  the  act  of  testing  the  models 


*To  test  and  evaluate  a  model  means  to  compare  its  performance  with  that  of  the  real 
world.  To  validate  a  model  means  that  the  model  was  accurate  enough  to  be  used  for  its 
designed  purpose.  The  effort  reported  here  emphasized  testing  and  evaluation. 
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can  reveal  the  strengths  of  the  new  models,  and  how  it  can  suggest  areas  where  further 
work  could  lead  to  improvements. 

For  readers  interested  in  performance,  the  report  contains  many  tables  that 
compare  the  predictions  of  the  models  documented  in  Carter  et  al.,  1987,  with  actual  loss 
rates  and  with  the  corresponding  loss  rates  estimated  by  the  model  currently  being  used 
by  the  Air  Force. 

For  readers  largely  interested  in  diagnostics,  the  same  tables  indicate  where  the 
models  are  weakest,  and  may  suggest  both  reasons  for  the  problems  and  ways  to  correct 
them. 

Readers  interested  in  technique  can  see  how  the  tables  were  calculated.  Both  the 
theory  and  some  of  the  detail  are  presented. 

OUTLINE 

The  basic  models  calculate  loss  rates  for  a  construct  called  a  year  at  risk.  Section 
II  defines  this  idea,  describes  how  estimated  loss  rates  were  compareu  with  actual  ones, 
and  presents  the  results. 

The  Air  Force  needs  loss  rates  for  fiscal  years,  and  the  process  of  calculating  fiscal 
year  loss  rates  using  year-at-risk  loss  rates  is  called  blending.  Section  III  describes  the 
theory  and  practice  of  blending  that  was  used  in  the  test  and  evaluation  and  compares  the 
blended  loss  rates  with  actual  ones. 

The  Air  Force  currently  uses  a  model  called  the  Airman  Loss  Probability  System 
(ALPS)  to  calculate  loss  rates,  and  the  middle-term  disaggregate  loss  models  described 
here  will  eventually  replace  ALPS.  Section  IV  compares  the  loss  rates  derived  from 
ALPS  and  the  new  models. 


II.  TEST  AND  EVALUATION  OF  THE  COHORT  MODELS 


INTRODUCTION 

These  models  were  estimated  with  a  40  percent  sample  of  airmen  who  were  in  the 
Air  Force  during  the  seven-year  period  from  fiscal  year  (FY)  1977  through  FY  1983. 

The  evaluation  described  here  used  10  percent  samples  of  airmen  who  were  in  the  Air 
Force  in  FY  83,  FY  84,  and  FY  85. 1  The  models  are  described  in  Carter  et  al.,  1987. 

These  results  adopt  a  "top  down"  approach,  beginning  at  the  most  general,  least 
detailed  level  by  comparing  the  performance  of  all  the  models  combined.  Successively 
less  general  and  more  detailed  levels  are  then  considered.  First  the  net  performance  of 
the  individual  models  are  compared,  then  the  performance  of  each  model  is  examined 
with  various  subgroups  of  airmen  to  whom  the  model  applies. 

All  of  the  models  make  loss-rate  predictions.  Several  of  the  models  also  make 
"extend-given-stay"  predictions.  Only  the  loss  rate  predictions  will  be  discussed  at  the 
general  level;  the  extend-given-stay  predictions  will  be  treated  in  the  sections  devoted  to 
the  individual  models. 

Cohort  Evaluation 

The  middle-term  disaggregate  loss  models  estimate  the  probability  of  loss  and  the 
probability  of  extending  given  nonloss  for  a  year  at  risk. 

The  term  year  at  risk  can  be  defined  recursively. 

1 .  Any  day  an  airman  enlists  or  reenlists  is  called  an  anniversary. 

2.  If,  during  the  year  following  an  anniversary,  the  airman  neither  reenlists  nor 
quits,  the  first  day  of  the  next  year  is  called  an  anniversary. 

3.  The  entire  year  following  any  anniversary  is  called  a  year  at  risk. 

Figure  1  illustrates  the  definitions  of  anniversary  and  year  at  risk.  The  symbol  "I" 

represents  an  anniversary,  and  the  symbol  "I - >"  represents  a  year  at  risk.  This 

airman  experienced  seven  anniversaries  and  seven  years  at  risk.  There  was  one  period 

!To  remain  consistent  with  the  estimation  procedures,  our  fiscal  years  begin  in  July,  so 
FY  83,  for  example,  begins  in  July  1982. 
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Airman  Airman  Airman 

enlists  reenlists  leaves 


Fig.  1 — Hypothetical  Airman  Career 


when  two  years  at  risk  overlapped,  because  he  reenlisted  for  a  second  term  before  the 
expiration  of  his  first  term.  Also,  for  the  last  part  of  his  last  year  at  risk  he  was  no  longer 
in  the  service. 

Every  airman  has  some  characteristics  that  remain  constant  from  one  year  at  risk 
to  the  next  (e.g.,  sex,  age  at  first  enlistment),  some  that  change  only  occasionally  (e.g., 
education,  marital  status),  and  some  that  may  change  often  (e.g.,  grade,  occupational 
class).  Each  year  at  risk  has  characteristics  of  its  own  (e.g.,  the  date  it  begins,  the  date  it 
ends,  the  unemployment  rate,  the  military/civilian  pay  ratio).  For  each  airman,  these  data 
can  be  assembled  into  a  rectangular  file  whose  rows  are  the  successive  years  at  risk 
experienced  by  that  airman.  The  airman  whose  career  is  illustrated  in  Fig.  1  would 
contribute  seven  records  to  this  file.  Merging  these  airman  files  over  the  entire  force  (or 
a  sample)  yields  the  Year-At-Risk  (YAR)  file,  which  was  used  both  to  estimate  and  to 
evaluate  the  loss  models. 

For  any  particular  airman,  a  given  day  might  lie  between  the  beginning  and  end  of 
one  or  more  of  his  years  at  risk;  when  this  happens  one  says  the  year  at  risk  covers  the 
day  in  question.  (More  than  one  year  at  risk  can  cover  a  particular  day  when  he  reenlists 
in  a  new  term  before  the  end  of  the  previous  one.)  Any  particular  day  defines  a  cohort  of 
airmen:  all  the  airmen  who  experienced  a  year  at  risk  that  covered  that  day.  This  Note 
discusses  the  loss  model  evaluation  that  was  performed  with  cohorts  defined  by  the 
beginning  of  fiscal  years  1983,  1984,  and  1985. 

Ten  loss  models  have  been  developed.  They  differ  according  to  the  the  term  of 
service  to  which  they  apply — first,  second,  career  (any  nonretirement  term  beyond  the 
second),  and  retirement.  Furthermore,  first,  second,  and  career  models  subdivide  into 
end  of  term  of  service  (ETS)  models  (years  at  risk  during  which  the  term  is  scheduled  to 
end),  attrition  models  (earlier  years  of  a  term),  and  extension  models  (later  years  of  a 
term).  The  10  models,  along  with  an  acronym  for  each,  are: 
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1  ATT 

1ETS 

1EXT 

2ATT 

2ETS 

2EXT 

CATT 

CETS 

CEXT 

RET 


First  term  attrition 
First  term  ETS 
First  term  extension 
Second  term  attrition 
Second  term  ETS 
Second  term  extension 
Career  attrition 
Career  ETS 
Career  extension 
Retirement 


Loss  predictions  for  a  cohort  of  airmen  over  some  year  at  risk  is  not  by  itself  of 
any  practical  interest.  What  is  important  is  the  estimation  of  the  size  of  the  force  at  the 
end  of  a  fiscal  year  (end  strength)  given  its  size  at  the  beginning.  During  any  given  fiscal 
year,  two  or  three  different  loss  models  may  be  needed  to  calculate  the  probability  that 
some  particular  airman  will  be  lost  during  that  fiscal  year.  The  technique  of  combining 
the  loss  models  to  make  such  a  calculation  is  called  blending.  Once  blending  is 
completed,  the  models  can  be  evaluated  by  seeing  how  closely  the  predicted  end  strength 
compares  with  the  actual:  Making  such  comparisons  is  called  fiscal  year  evaluation. 
Blending  and  fiscal  year  evaluation  are  discussed  in  Sec.  III. 

Evaluation  Measures 

This  section  will  define  several  measures  of  fit  illustrating  these  definitions  with  an 
example.  Using  all  10  models  to  predict  losses  for  a  10  percent  sample  of  airman  records 
from  the  YAR  file  (all  airmen  with  a  Social  Security  Number — SSN — that  ends  in  a  3) 
for  fiscal  year  1983,  yields: 


Number  of  airmen  at  risk 
Predicted  losses 
Actual  losses 


50,987 

5,686 

5,934 


Prediction  error  (PE)  is  simply  the  number  of  predicted  losses  minus  the  number 
of  actual  losses.  In  this  example,  the  prediction  error  is  -248.  The  size  of  the  prediction 
error  depends  partly  on  the  accuracy  of  the  model.  It  is  positive  if  the  model 
overpredicts  losses,  negative  if  the  model  underpredicts  losses.  But  its  size  also  depends 
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on  the  number  of  airmen  at  risk  of  loss;  and  unless  this  number  is  known,  prediction  error 
is  almost  meaningless. 

A  loss  prediction  also  implies  a  survival  (retention)  prediction.  The  survival 
prediction  error  (number  of  predicted  survivors  minus  the  number  of  actual  survivors)  is 
just  the  negative  of  the  prediction  error,  so  there  is  a  certain  symmetry  in  this  measure. 

Percent  Relative  Error  (PRE)  is  defined  to  be  the  prediction  error  divided  by  the 
number  of  actual  losses,  times  100.  In  the  example,  this  rate  is  -4.2  percent.  Since  these 
models  are  called  loss  models,  PRE  seems  to  be  a  natural  measure  of  performance. 
However,  it  has  some  peculiar  properties; 

1.  If  loss  rates  are  low,  the  PRE  can  be  quite  large,  even  though  the  prediction 
error  itself  is  small. 

2.  If  we  arc  not  interested  in  the  number  of  losses,  but  in  the  number  of 
survivors,  the  corresponding  error  rate  would  be  the  survival  prediction  error 
divided  by  the  number  of  survivors.  If  losses  are  low  and  the  number  of 
airmen  at  risk  is  high,  then  the  survival  PRE  will  be  small,  while  the  loss 
PRE  will  be  high.  In  this  example,  the  survival  PRE  is  0.55  percent,  an  order 
of  magnitude  lower  than  the  loss  PRE.  Thus,  PRE  is  an  ambiguous  measure 
of  performance,  unless  the  user  has  some  reason  to  value  accuracy  in  one 
prediction  (c.g.,  loss  prediction)  over  accuracy  in  the  other  (e.g.,  survival 
prediction). 

A  more  symmetric  measure  is  prediction  error  as  percent  of  number  at  risk 
(PEPNR),  defined  to  be  the  prediction  error  divided  by  the  number  of  airmen  at  risk, 
times  100.  In  this  example,  this  rate  is  -0.49  percent.  If  these  models  can  be  thought  of 
as  predicting  whether  an  airman  will  be  lost  during  an  applicable  year  at  risk,  PEPNR 
measures  the  net  error  rate — the  net  number  of  false  negatives  (if  PEPNR  is  negative)  or 
false  positives  (if  PEPNR  is  positive)  as  a  percent  of  the  number  of  predictions  made. 
PEPNR  for  survivals  is  just  the  negative  of  PEPNR  for  losses. 

These  three  measures  are  purely  descriptive  ones.  One  more  is  called 
standardized  prediction  error  ( SPE ).  Its  definition  requires  some  discussion. 

Each  middle-term  disaggregate  loss  model  is  a  linear  combination  of  the  form: 


LOSS  =  Constant 


+  Parameter^)  *  Variable(l) 

+  Parameter(2)  *  Variable(2) 

+ . . . 

+  Parameter(n)  *  Variable(n) 

Each  model  applies  to  individual  airmen.  In  any  particular  year  at  risk,  each 
airman  is  described  by  the  variables  in  the  model;  and  each  model  is  characterized  by  the 
variables  used  and  the  estimates  of  the  constant  and  the  parameters. 

Generally,  the  value  assigned  to  LOSS  is  a  number  that  lies  strictly  between  0  and 
1.  Because  an  airman  either  leaves  or  does  not,  LOSS  has  to  be  interpreted  as  a 
probability.  For  any  group  of  airmen,  the  values  assigned  to  LOSS  may  be  added  for  all 
airmen  in  the  group  to  predict  the  number  of  losses  within  this  group.  But  because  LOSS 
is  a  probability,  the  actual  number  of  losses  in  any  group  does  not  have  to  agree  exactly 
with  the  predicted  number.  Disagreement  between  the  predicted  number  and  actual 
number  of  losses  within  any  group  is  expected  and  does  not  necessarily  invalidate  the 
model. 

The  actual  number  of  losses  will  differ  from  the  predicted  number  by  chance 
alone,  even  if  the  model  represents  reality  perfectly.  It  may  even  differ  by  a  large 
amount,  but  as  the  difference  gets  larger,  the  probability  of  that  difference  gets  smaller. 

If  that  probability  is  small  enough,  something  must  be  wrong  with  the  model. 

Calculating  the  probability  of  observing  a  given  difference  between  the  actual  and 
predicted  number  of  losses  is  no  simple  matter.  The  main  problem  is  that  the  equations 
expressing  the  models  do  not  in  and  of  themselves  give  a  complete  description  of  what 
we  think,  or  at  least  what  we  are  willing  to  hypothesize.  The  two  main  problems  are 
homogeneity  and  independence. 

Homogeneity.  The  question  here  is  whether  a  model  assigns  each  airmen  his 
exact  loss  probability,  or  whether  it  merely  assigns  the  correct  loss  probability  to  some 
class  of  airman  to  which  that  airman  belongs. 

Most  of  the  variables  used  in  the  loss  models  are  categorical  ones — they  take  on 
just  a  few  values.  In  fact,  most  are  dummy  variables — variables  that  take  op  only  the 
values  0  or  1.  (The  two  important  exceptions  are  the  unemployment  rate  and  the 
militaiy/civilian  pay  ratio;  but  for  any  given  fiscal  year,  the  unemployment  rate  changes 
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very  little  and  the  pay  ratio  changes  not  at  all,  so  for  the  purposes  of  evaluation  these  two 
variables  can  be  regarded  as  part  of  the  constant  term.)  In  effect,  each  model  partitions 
the  force  into  a  small  number  of  categories  and  assigns  the  same  loss  probability  to  each 
airman  in  any  one  category.  The  question  of  homogeneity  is  whether  within  each  of 
these  categories  the  loss  probability  assigned  ro  each  airman  is  exactly  his  loss 
probability,  or  whether  it  is  simply  the  average  loss  probability  for  all  airman  in  that 
group.  If  the  former  is  true,  then  the  sample  is  homogeneous  (with  respect  to  this 
model). 

If  the  sample  is  homogeneous  with  respect  to  some  model,  it  is  possible  to 
calculate  the  probability  distribution  of  the  predicted  number  of  losses  in  any  particular 
sample  (but  see  the  next  section  on  independence)  and  then  to  assign  a  probability  to 
observing  any  specified  difference  between  actual  and  observed  losses. 

Of  course,  no  one  believes  our  samples  are  homogeneous.  In  any  stratum  defined 
by  these  models,  there  are  surely  other  variables  the  model  does  not  use,  perhaps  are  not 
even  contained  in  the  YAR  file,  that  would  partition  that  stratum  into  further  groups  of 
airmen  where  the  loss  probabilities  would  be  substantially  different  from  group  to  group. 
But  these  new  variables  are  unknown,  as  are  how  different  the  new  loss  probabilities 
would  be. 

Without  homogeneity,  probability  calculations  have  to  be  made  with  a 
homogeneity  assumption,  but  the  results  must  be  interpreted  with  the  understanding  that 
the  assumption  may  be  wrong. 

Independence.  The  question  here  is  whether  each  airman  leaves  the  force 
independently  of  the  choices  made  by  other  airmen.  An  assumption  about  independence 
is  required  to  calculate  the  distribution  of  the  actual  number  of  losses. 

If  the  sample  were  homogeneous  with  respect  to  the  model,  then  for  any  subgroup 
defined  by  the  model  whose  size  is  n  and  whose  predicted  loss  rate  is  p,  the  predicted 
number  of  losses  will  be  np.  If  within  this  group  the  airmen  act  independently,  the  actual 
number  of  losses  will  be  a  binomially  distributed  random  variable  with  mean  n*p  and 
variance  n*p*(l  -  p). 

At  another  extreme,  the  airmen  could  always  act  in  concert  (all  leave  together  or 
stay  together),  but  they  all  leave  with  probability  p  or  stay  with  probability  1  -  p.  This 
situation  could  still  be  appropriately  represented  by  one  of  these  models,  and  the  best 
prediction  of  the  number  of  losses  is  n*p.  The  average  number  of  airmen  lost  will  still  be 


n*p,  but  the  actual  number  will  be  a  random  variable  equal  to  n  with  probability  p  and  0 
with  probability  1  -  p,  and  its  variance  will  be  n  times  bigger  than  the  variance  of  the 
independent  case. 

As  a  third  alternative,  all  the  airmen  might  want  to  leave,  but  some  kind  of  policy 
constrains  things  so  that  once  the  fraction  p  has  left,  no  further  losses  are  allowed.  In 
other  words,  exactly  the  fraction  p  leaves,  no  more,  no  less.  The  models  still 
appropriately  represent  this  situation,  and  the  best  prediction  of  the  number  of  losses  is 
n*p.  In  that  case,  the  actual  number  of  losses  will  be  exactly  n*p,  the  variance  will  be 
zero,  and  our  predictions  will  always  be  perfect. 

Thus,  depending  on  the  degree  of  independence  of  choice  among  airmen  in  any 
homogeneous  group,  the  variance  of  the  actual  number  of  losses  can  be  quite  different. 

In  fact,  airmen  probably  do  not  act  independently.  Situations  similar  to  those 
mentioned  above,  though  not  as  extreme,  probably  occur.  Within  some  model  stratum, 
some  kinds  of  exogenous  events  might  occur  that  simultaneously  affect  the  decisions  of 
several  different  airmen.  After  a  certain  number  of  losses,  the  service  might  place  a 
block  in  front  of  any  further  losses  or  offer  some  additional  inducement  to  stay. 

The  statistical  measure  introduced  here  tests  the  hypothesis  that  our  predicted  loss 
rate  is  the  "true"  loss  rate,  under  the  assumptions  that  the  population  is  homogeneous 
with  respect  to  the  model,  and  that  airmen  act  independently.  Under  this  hypothesis,  the 
number  of  losses  in  any  group  will  be  a  random  variable  whose  mean  m  is 

m  =  N*p 

and  whose  variance  v  is  larger  than 

v  =  N*p*(l  -  p) 
where 

N  =  number  of  airmen  at  risk  in  the  group 
p  =  predicted  loss  rate  for  the  entire  group 

If  N  is  large  enough,  under  this  hypothesis  the  standardized  prediction  error 


SPE  =  PE/sqrt(v), 


is  approximately  normally  distributed  with  variance  less  than  1.  This  means  that  the 
probability  of  observing  a  given  prediction  error  can  be  estimated  from  tables  of  the 
standard  normal  distribution 

Prob(ISPEI  >  z)  >  Prob(IZI  >  z). 

Generally,  if  the  absolute  value  of  SPE  exceeds  2,  the  hypothesis  of  a  homogeneous 
population  of  independently  acting  airmen  with  the  given  loss  rate  can  be  rejected. 

For  the  example,  SPE  is  3.5,  which  means  the  hypothesis  of  a  perfect  statistical  fit 
can  be  rejected.  Of  course,  the  sample  is  not  homogeneous,  or  there  would  not  be  so 
many  models.  However,  this  fact  alone  probably  does  not  explain  the  lack  of  fit.  As  will 
be  seen  in  Sec.  Ill,  several  of  the  individual  models  do  not  fit  the  data  well.  But  more 
important,  with  large  sample  sizes,  statistical  models  rarely  fit  the  data,  especially  when 
the  models  attempt  to  predict  human  behavior.  Given  the  complexity  of  the  process 
modelled  here,  and  the  size  of  the  sample  (over  50,000  cases),  a  3.5  standard  deviation 
error  is  probably  not  too  bad. 

COMPARING  THE  MODELS 

The  "top  down"  approach  to  describing  the  evaluation  results  first  consider  briefly 
what  happens  when  the  predictions  of  all  10  models  are  combined  to  estimate  the  net 
number  of  losses  for  the  entire  sample.  The  predictions  of  the  10  models  are  compared 
with  actual  losses,  first  for  fiscal  year  1983  and  then  for  fiscal  years  1984  and  1985. 

Fiscal  year  1983  was  used  for  estimation.  There  are  two  reasons  to  use  it  for 
evaluation  as  well.  First,  it  verified  the  implementation  of  the  loss  models  in  the  form  of 
executable  code.  Making  calculations  for  1983  and  seeing  that  the  predicted  and  actual 
loss  rates  are  close  helps  to  confirm  that  the  implementation  is  correct.  Second,  even 
though  1983  data  were  among  those  used  to  estimate  the  models,  there  is  no  guarantee 
that  the  models  will  necessarily  perform  well  for  1983.  Loss  rates  in  1983  might  be  quite 
different  from  those  in  previous  years,  and  the  model  will  predict  average  loss  rates 
perfectly  only  if  the  averages  are  taken  from  the  entire  seven-year  period.  Using  1983  by 
itself  is  therefore  a  useful  test  of  the  model. 


Comparisons  When  Loss  Rates  Are  Combined 

The  overall  performance  of  the  10  models  for  the  three  years  is  as  shown  in  Table 

2.1. 

This  display  suggests  that  the  overall  performance  of  the  models  is  quite  good. 

The  relative  prediction  error  is  at  most  5.6  percent  (in  FY  85,  which  is  two  years  beyond 
the  data  used  to  estimate  the  model).  But  'wen  in  FY  85,  the  models  taken  together 
predict  only  333  losses  too  many,  out  of  nearly  52,000  airmen  in  the  sample.  The 
standardized  prediction  error  has  increased  (in  absolute  value)  by  1  standard  deviation 
over  the  level  in  FY  83. 

Comparing  the  Models  for  Fiscal  Year  1983 

Table  2.2  summarizes  the  performance  of  each  of  the  10  models  for  FY  83. 

Recall  that  data  for  1983  were  used  to  estimate  the  coefficients  of  these  models,  so  good 
performance  for  this  year’s  data  would  be  expected  if  the  evaluation  code  were  correctly 
written. 

Four  of  the  models,  2ATT,  2ETS,  CATT,  and  CEXT,  exhibit  large  relative  errors. 
However,  in  the  case  of  CEXT  the  number  of  losses  is  low,  and  because  the  standardized 
prediction  error  is  low  the  prediction  error  could  be  due  to  chance  alone. 

The  second-term  and  career  attrition  models  exhibit  errors  that  cannot  be  so  easily 
dismissed.  In  each  case,  the  actual  number  of  losses  is  more  than  3.5  standard  deviations 
from  the  predicted  number.  The  error  seen  in  the  second-term  ETS  model  is  large 
enough  to  be  noted,  but  not  large  enough  to  be  alarming.  Predicted  losses  exceed  actuals 
by  a  little  over  two  standard  deviations.  Although  the  probability  of  such  an  event  is 

Tabic  2.1 

OVERALL  PERFORMANCE,  10  MODELS,  FY  83-86 


Item 

FY  83 

FY  84 

FY  85 

Number  of  airmen  at  risk 

50,987 

51,769 

51,933 

Predicted  losses 

5,686 

5,567 

6,299 

Actual  losses 

5,934 

5,592 

5,966 

Prediction  error  (PE) 

-248 

-25 

333 

Percent  relative  error  (PRE) 

-4.2 

-0.4 

5.6 

Error  as  %  of  N  at  risk  (PEPNR) 

-0.5 

-0.1 

0.6 

Standardized  prediction  error  (SPE) 

-3.5 

-0.4 

4.5 

Table  2.2 

PERFORMANCE  OF  ALL  LOSS  MODELS  FOR  FY  83 


Model 

Airmen 

at 

Risk 

Loss  Rates  (<% ) 

Predicted  Actual 

PE 

PRE (9c) 

PEPNR  (9C) 

SPE 

1  ATT 

19461 

9.57 

9.95 

-74 

-3.8 

-0.38 

-1.80 

1ETS 

4283 

34.47 

35.86 

-60 

-3.9 

-1.39 

-1.91 

IE  XT 

2850 

30.81 

31.23 

-12 

-1.3 

-0.42 

-0.49 

2ATT 

7117 

1.98 

2.61 

-45 

-24.1 

-0.63 

-.3.82 

2ETS 

1203 

14.26 

12.05 

27 

18.3 

2.21 

2.19 

2EXT 

1037 

14.71 

14.27 

5 

3.1 

0.44 

0.40 

CATT 

9169 

0.68 

1.06 

-35 

-35.8 

-0.38 

-4.43 

CETS 

1617 

1.76 

1.67 

1 

5.4 

0.09 

0.28 

CEXT 

829 

3.77 

4.34 

-5 

-13.1 

-0.57 

-0.86 

RET 

3421 

25.78 

27.27 

-51 

-5.5 

-1.49 

- 1 .99 

Total 

50987 

11.15 

11.63 

-248 

-4.2 

-0.49 

-3.50 

under  5  percent,  the  probability  of  its  occurrence  in  10  trials  exceeds  20  percent,  so  it  is 
not  surprising  to  see  a  difference  this  large  at  least  once  in  a  list  of  this  sort. 

Comparing  the  Models  for  Fiscal  Years  1984  and  1985 

Table  2.3  compares  actual  and  predicted  loss  rates  for  all  three  fiscal  years.  Data 
for  FY  84  and  FY  85  were  not  used  to  estimate  the  parameters  of  these  models,  so  Table 
2.3  represents  a  more  severe  test  than  does  Table  2.2. 

Considering  standardized  prediction  error  only,  six  of  the  models  (1ATT,  1EXT, 
2EXT,  CETS,  CEXT,  and  RET)  perform  exactly  as  one  would  expect  an  unbiased  model 
to  behave.  Of  these  six  models  and  three  fiscal  years,  there  is  only  one  example  where 
the  SPE  exceeds  2  (2 EXT  in  FY  85). 

As  in  FY  83,  2ATT,  2ETS,  CATT,  and  CEXT  exhibit  large  PREs  in  both  FY  84 
and  FY  85.  As  in  FY  83,  CEXT  loss  rates  are  extremely  low,  and  because  the 
standardized  prediction  error  is  small,  the  eno.  0een  in  that  modci  is  no  cause  for 
concern. 

The  two  attrition  models  2ATT  and  CATT  continue  to  display  the  large  SPEs  seen 
in  FY  83,  suggesting  something  may  be  wrong  with  either  the  specification  or  the 
implementation  of  these  two  models.  In  both  models,  the  problem  lies  in  the  failure  of 
the  models  to  predict  big  changes  in  the  actual  loss  rates;  in  other  words,  the  models  are 
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Airmen  Loss  Rates  (%) 


at 


Model 

FY 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR(%) 

SPE 

1  ATT 

83 

19461 

9.57 

9.95 

-74 

-3.8 

-0.4 

-1.80 

84 

19043 

9.04 

9.00 

8 

0.4 

0.0 

0.19 

85 

18301 

9.07 

8.69 

70 

4.2 

0.4 

1.79 

1ETS 

83 

4283 

34.47 

35.86 

-60 

-3.9 

-1.4 

-1.91 

unresponsive  rather  than  overreactivc.  Of  course,  an  error  in  the  estimate  of  the  actual 
loss  rate  is  another  possible  reason  for  the  differences. 

The  second-term  ETS  model  exhibits  the  same  marginal  performance  seen  in  FY 
83,  with  standardized  prediction  errors  a  little  too  large  to  be  satisfactory,  especially 
when  seen  in  three  successive  fiscal  years. 
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Models  for  Extend-GIven-Stay  Rates 

In  addition  to  estimating  loss  rates,  each  of  the  ETS  models  estimates  the  rate  at 
which  airmen  extend  their  terms,  given  they  do  not  leave.  This  rate  is  the  extend- given- 
stay  rate. 

Naturally  enough,  these  models  were  estimated  using  the  subsample  of  airmen 
who  did  not  leave  the  service  at  the  end  of  the  term  year.  Unfortunately,  the  models 
cannot  be  applied  to  exactly  this  group  of  airmen.  The  loss  models  do  not  predict  who 
will  leave,  they  only  assign  a  probability  of  leaving  to  each  airman.  After  this  probability 
is  assigned,  there  is  no  subset  of  airmen  that  can  be  designated  as  "having  stayed"  to 
which  the  extend-given-stay  model  may  be  applied.  Consequently,  to  test  these  models 
the  extend-given-stay  rate  was  calculated  for  every  airman.  This  was  compared  with  an 
"actual"  rate  obtained  by  dividing  the  number  of  airmen  who  extended  by  the  number 
who  did  not  leave.  This  causes  another  problem:  The  samples  used  to  calculate  a 
predicted  rate  and  an  actual  rate  are  different;  the  sample  used  to  calculate  the  actual  rate 
is  a  subset  of  the  sample  used  to  calculate  the  predicted  rate.  To  make  the  same  kinds  of 
comparisons  (prediction  error,  percent  relative  error,  etc.)  as  made  for  loss  rates,  the 
number  of  airmen  at  risk  is  taken  for  the  number  that  actually  stayed,  even  though  the 
prediction  was  based  on  a  larger  sample. 


Table  2.4 

PERFORMANCE  OF  EXTEND-GIVEN-STAY  MODELS,  FY  83,  FY  84,  FY  85 


Term 

FY 

Airmen 

at 

Risk 

Extend-Givcn- 
Stay  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

1st 

83 

2747 

48.08 

48.34 

-7 

-0.5 

-0.2 

-0.22 

84 

2755 

48.63 

41.45 

198 

14.8 

4.4 

5.89 

85 

3073 

47.84 

39.86 

245 

16.7 

5.0 

6.98 

2nd 

83 

1058 

40.94 

36.11 

51 

11.8 

4.2 

3.00 

84 

1223 

41.24 

36.39 

59 

11.8 

4.2 

3.21 

85 

1150 

49.75 

41.22 

98 

17.1 

7.1 

5.26 

Career 

83 

1590 

22.50 

28.99 

-103 

-28.8 

-6.4 

-6.15 

84 

1700 

21.93 

28.53 

-112 

-30.1 

-6.4 

-6.50 

85 

1591 

21.70 

29.16 

-119 

-34.4 

-7.2 

-7.11 

-15- 
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Table  2.4  compares  actual  and  predicted  extend-givcn-stay  rates  for  all  three 
models  and  all  three  fiscal  years  used  in  the  evaluation.  As  can  be  seen,  the  models  do 
not  perform  as  well  as  the  loss  models. 

DETAILED  DISCUSSION  OF  THE  MODELS 

For  each  of  the  10  models,  and  for  each  fiscal  year  (1983,  1984,  and  1985), 
predicted  and  actual  loss  rates  were  calculated  for  all  airmen  in  the  10  percent  YAR 
sample  designated  SSN3  (Carter  et  al.,  1987).  To  evaluate  the  performance  of  the 
models  over  different  groups  of  airmen,  these  samples  were  also  stratified  with  seven 
factors  and  loss  rates  calculated  for  all  airmen  within  the  strata  defined  by  these  factors. 
The  seven  factors  used  were: 

Occupational  group 
Year  of  term 
Bonus  multiple 
Grade 

Term  of  enlistment 

Decision  status  (extension  models) 

Missing  values 

The  models  apply  to  airman  samples  ranging  from  around  20,000  cases  (first-tcim 
attrition)  to  around  1,000  cases  (second-term  extension).  When  the  samples  were 
stratified  with  respect  to  one  or  another  of  the  above  factors,  some  of  the  resulting  strata 
were  too  small  to  provide  interesting  information  about  the  model’s  performance. 

Results  for  such  small  strata  are  not  reported. 

First-Term  Attrition 

The  first-term  attrition  model2  applies  to  approximately  20,000  airmen,  about  40 
percent  of  the  sample.  We  examine  the  performance  of  this  model  using  two  strata: 
term  year  (Table  2.5)  and  Career  Field  Group  (Table  2. 6). 3  Because  many  airmen  do  not 

2Carter  et  al.  (1987),  Table  2.2  and  Table  2.3. 

3Carecr  Field  Groups  (CFGs)  arc  defined  in  Table  A.l  of  Carter  et  al.  (1987).  They 
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Electrical/Mechanical  Equipment  Repairmen 
Functional  Support  and  Administration 
Craftsmen,  Service  and  Supply  Handlers 
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appear  to  receive  occupational  classifications  until  their  third  year,  the  latter  table  is 
restricted  to  such  airmen. 

This  model’s  performance  is  generally  good.  The  largest  standardized  prediction 
error  for  a  single  fiscal  year  occurred  in  FT  83  but  was  within  acceptable  limits, 
underpredicting  the  number  of  losses  by  only  74  airmen  out  of  a  total  of  over  19,000  at 
risk.  During  that  year,  the  predicted  loss  rate  practically  coincided  with  the  actual  rate 
for  first-year  airmen,  a  good  sign  because  the  submodel  for  first-year  airmen  is  a  bit 
complicated.  In  FY  84  the  overall  predicted  loss  rate  almost  coincides  with  the  actual, 
but  within  term  years  the  model  overpredicts  loss  rates  for  first-  and  second-year  airmen 
and  underpredicts  such  losses  for  airmen  in  their  third  year.  This  underprediction  is  the 
shaipest  discrepancy  seen  in  these  tables,  but  no  such  error  occurs  in  FY  85,  where  in 
fact  the  model  predicts  losses  quite  accurately. 

Errors  by  CFGs  for  the  most  part  fall  within  acceptable  bounds.  Predicted  loss 
rates  in  FY  84  are  low  for  third-year  airmen,  and  this  underprediction  is  seen  consistently 
for  all  four  CFGs  that  year.  Whatever  happened  to  affect  the  overall  loss  rate  for  those 
airmen  affected  them  consistently  with  respect  to  CFG. 


Table  2.5 


FIRST-TERM  ATTRITION  LOSSES  BY  TERM  YEAR 


FY 

Term  Year 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

19461 

9.57 

9.95 

-74 

-4.0 

-0.4 

-1.80 

1 

7188 

11.49 

11.48 

1 

0.1 

0.0 

0.03 

2 

6470 

8.45 

8.92 

-30 

-5.6 

-0.5 

-1.36 

3 

5528 

8.43 

9.33 

-50 

-10.7 

-0.9 

-2.41 

84 

All 

19043 

9.04 

9.00 

8 

0.4 

0.0 

0.19 

1 

6151 

11.04 

9.40 

101 

14.9 

1.6 

4.10 

2 

6362 

8.95 

8.16 

50 

8.8 

0.8 

2.21 

3 

5886 

7.10 

9.43 

-137 

-32.8 

-2.3 

-6.96 

85 

All 

18301 

9.07 

8.69 

70 

4.2 

0.4 

1.79 

1 

5994 

10.62 

10.63 

-1 

-0.1 

-0.0 

-0.03 

2 

5569 

8.97 

7.09 

105 

21.0 

1.9 

4.91 

3 

5843 

7.75 

7.94 

-11 

-2.5 

-0.2 

-0.54 

Table  2.6 


FIRST-TERM  ATTRITION  LOSSES  BY  CAREER  FIELD  GROUP 

(Third  year) 


Airmen  Loss  Rates  (%) 


FY 

Career 
Field  Group 

at 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR  (%) 

S  PE 

83 

All 

5528 

8.43 

9.33 

-50 

-10.7 

-0.9 

-2.41 

SkilTech 

1744 

7.29 

8.14 

-15 

-11.7 

-0.8 

-1.37 

ElMeRepa 

1641 

8.20 

10.05 

-30 

-22.6 

-1.9 

-2.73 

Support 

1159 

8.05 

8.80 

-9 

-9.3 

-0.7 

-0.94 

CrfSerSu 

925 

11.49 

10.81 

6 

5.9 

0.7 

0.65 

84 

All 

5886 

7.10 

9.43 

-137 

-32.8 

-2.3 

-6.96 

SkilTech 

1500 

6.46 

9.27 

-42 

-43.5 

-2.8 

-4.43 

ElMeRepa 

1601 

7.65 

10.18 

-41 

-33.1 

-2.5 

-3.81 

Support 

806 

7.90 

12.78 

-39 

-61.8 

-4.9 

-5.14 

CrfSerSu 

730 

11.15 

16.03 

-36 

-43.8 

-4.9 

-4.19 

85 

All 

5843 

7.75 

7.94 

-11 

-2.5 

-0.2 

-0.54 

SkilTech 

2084 

5.93 

6.48 

-11 

-9.3 

-0.6 

-1.06 

ElMeRepa 

1848 

7.71 

7.90 

-4 

-2.5 

-0.2 

-0.31 

Support 

840 

7.89 

9.52 

-14 

-20.7 

-1.6 

-1.75 

CrfSerSu 

1055 

11.25 

9.48 

19 

15.7 

1.8 

1.82 

First-Term  ETS 

The  first-term  ETS  model  has  two  parts.  One  part  calculates  loss  rates;  the  other, 
extend-given-stay  rates. 

First-Term  ETS  Losses.  The  first-term  ETS  loss  model4  applies  to 
approximately  4,000  airmen,  or  about  8  percent  of  the  entire  sample.  The  performance 
of  this  model  is  examined  with  respect  to  CFG  (Table  2.7),  bonus  (Table  2.8),  and  the 
presence  or  absence  of  certain  variables  needed  by  the  model  (Table  2.9). 

Although  this  model  does  a  good  job  in  FY  83,  its  performance  deteriorates  in  FY 
84,  and  in  FY  85  it  sharply  overpredicts  actual  losses.  In  FY  85,  for  example,  it  predicts 
nearly  500  more  losses  than  were  actually  observed,  out  of  approximately  5,000  airmen 
at  risk.  This  overprediction  is  seen  in  all  four  CFGs,  and  is  unaffected  by  presence  or 
absence  of  a  bonus. 

4Carter  et  al.  (1987),  Table  5.2. 


One  problem  with  testing  the  first-term  ETS  model  is  the  presence  of  many  cases 
in  the  YAR  file  with  missing  data.  There  are  two  causes.  First,  the  model  uses  the 
airman’s  Air  Force  Specialty  Code  (AFSC)  code  to  look  up  the  value  of  its  cross-bonus 
variable.5  But  for  many  airmen,  the  required  value  could  not  be  found.  Second,  in 
several  cases  the  value  of  the  airman’s  bonus  coefficient  was  unknown. 

Cases  with  missing  data  were  given  a  default  value  for  the  missing  variable. 
Because  the  first-term  ETS  model  was  the  only  one  with  such  a  large  number  of  such 
missing  cases,  these  missing  cases  might  account  for  the  problems  described  above. 
Table  2.9  shows  the  model’s  performance  for  that  subset  of  cases  where  no  missing 
valu 's  were  found,  for  those  cases  where  only  the  cross-bonus  variable  (WBONC)  was 
missing,  and  for  those  cases  where  both  WBONC  and  the  bonus  coefficient  were 
missing.  In  FY  85,  the  problem  year,  the  model  performs  better,  if  anything,  on  the  cases 
with  missing  data  than  on  the  cases  with  no  missing  data.  The  prediction  errors  are 
therefore  not  caused  by  the  missing  values.6 

Extend-Given-Stay  Rates.  In  our  sample,  the  first-term  ETS  model  applies  to 
around  4,000  airmen,  with  a  loss  rate  of  around  35  percent,  hence  approximately  2,700 
airmen  remain  and  are  faced  with  the  "extend-given-stay"  decision.  The  performance  of 

Table  2.9 

FIRST-TERM  ETS  LOSSES  BY  WHAT’S  MISSING 


FY 

What’s 

Missing 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

Nothing 

3577 

34.45 

35.87 

-51 

-4.1 

-1.4 

-1.79 

WBONC 

234 

36.48 

37.61 

-3 

-3.1 

-1.1 

-0.36 

WBONC+BONUS 

465 

33.64 

64.62 

-144 

-92.1 

-31.0 

-14.14 

84 

Nothing 

3736 

38.69 

36.51 

81 

5.6 

2.2 

2.74 

WBONC 

226 

40.09 

34.07 

14 

15.0 

6.0 

1.85 

WBONC+BONUS 

521 

40.15 

38.00 

11 

5.4 

2.2 

1.00 

85 

Nothing 

3072 

46.96 

37.24 

299 

20.7 

9.7 

10.79 

WBONC 

1375 

48.52 

37.89 

146 

21.9 

10.6 

7.89 

WBONC+BONUS 

497 

50.21 

41.85 

42 

16.7 

8.4 

3.73 

5Carter  et  al.  (1987),  Table  5.2. 

6This  is  one  of  several  models  that  have  been  rcestimated  using  a  different 
mathematical  specification  as  a  result  of  the  test  and  evaluation  process. 


this  model7  is  examined  with  respect  to  Career  Field  Group  (Table  2. 10),  bonus  (Table 
2.11),  and  missing  values  (Table  2.12). 

The  model  performs  extremely  well  in  FY  83  but  poorly  in  both  FY  84  and  FY  85. 
In  general,  the  overall  extend-given-stay  rate  dropped  sharply  in  FY  84  and  remained 
there  in  FY  85,  and  the  model  was  unable  to  follow  this  drop.  Some  subgroups  where  the 
rate  was  underpredicted  in  FY  83  (e.g.,  the  CFG  "Support")  did  better  in  FY  84  and  FY 
85,  not  because  the  rates  fell  in  these  groups,  but  because  they  were  already  low. 

There  is  no  evidence  that  the  missing  values  cause  the  low  predicted  rates.  The 
fact  that  the  standardized  prediction  error  is  lower  for  the  missing  cases  simply  reflects 
the  sample  sizes  being  smaller. 

First-Term  Extension 

The  first-term  extension  model8  applied  to  almost  3,000  airmen  in  FY  83,  about  6 
percent  of  the  sample.  By  FY  85  the  number  had  shrunk  to  around  1,700  airmen,  about 
3.4  percent  of  the  sample.  The  performance  of  this  model  is  examined  with  respect  to 


Table  2.10 

FIRST-TERM  EXTEND-GIVEN-STAY  BY  CAREER  FIELD  GROUP 


Airmen  Stay  Rates 


FY 

Career 
Field  Group 

at 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

2747 

48.08 

48.34 

-7 

-0.5 

-0.2 

-0.22 

SkilTech 

868 

46.56 

47.12 

-5 

-1.2 

-0.3 

-0.26 

ElMeRepa 

794 

45.47 

41.69 

30 

8.3 

2.5 

1.73 

Support 

660 

48.84 

52.27 

-23 

-7.0 

-2.5 

-1.50 

CrfScrSu 

411 

54.32 

57.91 

-15 

-6.6 

-2.0 

-1.10 

84 

All 

2755 

48.63 

41.45 

198 

14.8 

4.4 

5.89 

SkilTech 

871 

48.26 

38.35 

86 

20.5 

6.2 

4.62 

ElMeRepa 

840 

46.47 

37.02 

79 

20.3 

5.9 

4.34 

Support 

698 

48.69 

45.85 

20 

5.8 

1.9 

1.23 

CrfSerSu 

427 

53.56 

48.24 

23 

9.9 

3.3 

1.74 

85 

All 

3073 

47.84 

39.86 

245 

16.7 

5.0 

6.98 

SkilTech 

959 

47.73 

39.21 

82 

17.9 

5.1 

4.07 

ElMeRepa 

1120 

46.05 

41.07 

56 

10.8 

3.1 

2.63 

Support 

614 

48.72 

34.04 

90 

30.1 

10.3 

6.10 

CrfScrSu 

373 

51.95 

47.18 

18 

9.2 

2.8 

1.41 

7Carter  et  al.  (1987),  Table  5.2. 

8Carter  ct  al.  (1987),  Table  8. 1  and  Table  8.2. 
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Table  2,11 


FIRST-TERM  EXTEND-GIVEN-STAY  BY  BONUS 


FY 

Bonus 

Airmen 

at 

Risk 

Stay  Rates 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

2747 

48.08 

48.34 

-7 

-0.5 

-0.2 

-0.22 

None 

1544 

50.44 

51.88 

-22 

-2.9 

-1.0 

-0.92 

1 

749 

45.14 

45.66 

-4 

-1.2 

-0.3 

-0.23 

84 

All 

2755 

48.63 

41.45 

198 

14.8 

4.4 

5.89 

None 

1764 

49.66 

44.44 

92 

10.5 

3.4 

3.53 

1 

396 

46.50 

33.08 

53 

28.9 

8.4 

4.25 

85 

All 

3073 

47.84 

39.86 

245 

16.7 

5.0 

6.98 

None 

1771 

49.16 

41.39 

138 

15.8 

5.0 

5.25 

1 

431 

45.79 

36.19 

41 

21.0 

6.0 

3.17 

Table  2.12 


FIRST-TERM  EXTEND-GIVEN-STAY  BY  MISSING  VALUES 


FY 

Missing 

Airmen 

at 

Risk 

Stay  Rates 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

2747 

48.08 

48.34 

-7 

-0.5 

-0.2 

-0.22 

None 

2294 

48.89 

49.26 

-8 

-0.8 

-0.2 

-0.28 

WBONC 

146 

45.02 

36.30 

13 

19.4 

5.4 

1.67 

WBONC+BONUS 

304 

43.42 

46.71 

-10 

-7.6 

-2.2 

-0.94 

84 

All 

2755 

48.63 

41.45 

198 

14.8 

4.4 

5.89 

None 

2372 

49.34 

42.16 

170 

14.6 

4.6 

5.57 

WBONC 

149 

46.82 

41.61 

8 

11.1 

3.4 

1.03 

WBONC+BONUS 

323 

44.57 

34.67 

32 

22.2 

6.1 

2.82 

85 

All 

3073 

47.84 

39.86 

245 

16.7 

5.0 

6.98 

None 

1928 

48.53 

38.59 

192 

20.5 

6.2 

6.92 

WBONC 

854 

47.48 

42.51 

42 

10.5 

3.1 

2.29 

WBONC+BONUS 

289 

44.56 

40.48 

12 

9.2 

2.4 

1.06 
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Career  Field  Group  (Table  2.13),  bonus  (Table  2.14),  grade  (Table  2.15),  and  whether 
the  airman  was  in  a  decision  year  (Table  2.16). 

This  model  seems  to  be  rather  good.  The  standardized  prediction  error  rarely 
exceeds  1  in  absolute  value  in  any  fiscal  year  and  for  any  stratum.  Curiously,  the  model 
does  best  in  FY  85,  two  years  beyond  the  data  used  to  estimate  it.  Extension  models 


Table  2. 13 

FIRST-TERM  EXTENSION  LOSSES  BY  CAREER  FIELD  GROUP 


Airmen  Loss  Rates  (%) 


FY 

Career 
Field  Group 

at 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

2850 

30.81 

31.23 

-12 

-1.4 

-0.4 

-0.49 

SkilTech 

871 

35.67 

33.87 

16 

5.0 

1.8 

1.11 

ElMeRepa 

832 

30.83 

31.25 

-3 

-1.4 

-0.4 

-0.26 

Support 

631 

25.90 

30.11 

-27 

-16.3 

-4.2 

-2.41 

CrfSerSu 

491 

27.59 

27.29 

1 

1.1 

0.3 

0.15 

84 

All 

2076 

30.95 

30.49 

10 

1.5 

0.5 

0.45 

SkilTech 

643 

35.69 

37.48 

-12 

-5.0 

-1.8 

-0.95 

ElMeRepa 

503 

30.40 

27.24 

16 

10.4 

3.2 

1.54 

Support 

540 

26.79 

28.33 

-8 

-5.7 

-1.5 

-0.81 

CrfSerSu 

371 

28.39 

26.15 

8 

7.9 

2.2 

0.96 

85 

All 

1706 

32.75 

31.65 

19 

3.4 

1.1 

0.97 

SkilTech 

491 

39.58 

37.88 

8 

4.3 

1.7 

0.77 

ElMeRepa 

433 

30.84 

31.41 

-2 

-1.8 

-0.6 

-0.26 

Support 

470 

27.26 

25.96 

6 

4.8 

1.3 

0.63 

CrfSerSu 

302 

32.44 

30.79 

5 

5.1 

1.6 

0.61 

Table  2.14 

FIRST-TERM  EXTENSION  LOSSES  BY  BONUS 


FY 

Bonus 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

2850 

30.81 

31.23 

-12 

-1.4 

-0.4 

-0.49 

None 

1769 

29.92 

31.03 

-20 

-3.7 

-l.i 

-1.02 

1 

884 

31.06 

31.00 

1 

0.2 

0.1 

0.04 

84 

All 

2076 

30.95 

30.49 

10 

1.5 

0.5 

0.45 

None 

1287 

30.04 

31.08 

-13 

-3.5 

-1.0 

-0.81 

1 

557 

31.48 

27.47 

22 

12.7 

4.0 

2.04 

85 

Ail 

1706 

32.75 

31.65 

19 

3.4 

1.1 

0.97 

None 

1129 

30.65 

29.23 

16 

4.6 

1.4 

1.03 

1 

252 

39.55 

39.65 

-0 

-0.3 

-0.1 

-0.03 

i  *  ■  9 
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Table  2.15 

FIRST-TERM  EXTENSION  LOSSES  BY  GRADE 


FY 

Grade 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

2850 

30.81 

31.23 

-12 

-1.4 

-0.4 

-0.49 

E-4 

2124 

30.44 

30.04 

8 

1.3 

0.4 

0.40 

E-5 

608 

31.41 

30.59 

5 

2.6 

0.8 

0.44 

84 

All 

2076 

30.95 

30.49 

10 

1.5 

0.5 

0.45 

E-4 

1275 

26.58 

26.04 

7 

2.0 

0.5 

0.44 

E-5 

665 

38.05 

40.00 

-13 

-5.1 

-2.0 

-1.04 

85 

All 

1706 

32.75 

31.65 

19 

3.4 

1.1 

0.97 

E-4 

1346 

31.33 

30.65 

9 

2.2 

0.7 

0.54 

E-5 

275 

37.13 

28.73 

23 

22.6 

8.4 

2.88 

contain  two  submodels,  one  that  applies  to  airmen  in  the  first  year  of  a  two-year 
extension,  where  such  losses  as  may  occur  correspond  to  attrition  losses,  and  another 
model  that  applies  to  airmen  in  the  last  year  of  an  extension.  Table  2.16  compares  the 
performance  of  the  first-term  extension  model  by  the  decision  status. 

The  loss  rates  are  very  different  for  decisionmakers  and  nondecisionmakers.  But 
the  fraction  of  all  airmen  at  risk  who  are  decisionmakers  can  change  quite  substantially 

Table  2.16 

FIRST-TERM  EXTENSION  LOSSES  BY  DECISIONMAKER  STATUS 
Airmen  Loss  Rate  (%) 


FY 

Attrition 

/Decision 

at 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

2850 

30.81 

31.23 

-12 

-1.4 

-0.4 

-0.49 

Attrition 

844 

5.01 

5.09 

-1 

-1.6 

-0.1 

-0.11 

Decision 

2006 

41.66 

42.22 

-11 

-1.3 

-0.6 

-0.51 

84 

All 

2076 

30.95 

30.49 

10 

1.5 

0.5 

0.45 

Attrition 

606 

5.03 

6.11 

-7 

-21.5 

-1.1 

-1.22 

Decision 

1470 

41.63 

40.54 

16 

2.6 

1.1 

0.85 

85 

All 

1706 

32.75 

31.65 

19 

3.4 

1.1 

0.97 

Attrition 

660 

5.02 

5.61 

-11.8 

-0.6 

-0.69 

Decision 

1046 

50.25 

48.09 

23 

4.3 

2.2 

1.40 
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from  one  year  to  the  next.  Ir  FY  83  decisionmakers  made  up  about  70  percent  of  the 
airmen  at  risk,  while  in  FY  85  they  made  up  about  61  percent.  A  decrease  in  the 
decisionmaker  fraction  should  lead  to  a  decrease  in  the  overall  loss  rate.  However, 
between  FY  83  and  FY  85  the  loss  rates  among  decisionmakers  increased,  and  so  for  the 
entire  group,  decisionmakers  and  nondecisionmakers,  about  the  same  fraction  was  lost  in 
each  fiscal  year.  The  model  was  able  to  track  these  countervailing  tendencies. 

Second-Term  Attrition 

The  second-term  attrition  model9  applied  to  a  little  over  7,000  airmen  in  FY  83, 
about  14  percent  of  the  force.  By  FY  85  this  number  had  increased  to  a  little  over  9,000 
and  constituted  about  18  percent  of  the  force.  The  performance  of  this  model  is 
examined  with  respect  to  term  year  (Table  2.17),  Career  Field  Group  (Table  2.18),  bonus 
(Table  2.19),  grade  (Table  2.20),  and  term  of  enlistment — TOE — (Table  2.21). 

The  model  underpredicted  loss  rates  slightly  in  FY  83;  but  loss  rates  were  low,  so 
the  number  of  errors  as  a  percent  of  the  number  of  airmen  at  risk  was  also  low.  If  the 
model  had  maintained  this  level  of  performance  in  subsequent  years  its  performance 
would  have  been  fair.  However,  actual  loss  rates  increased  sharply  in  both  FY  84  and 

Table  2.17 

SECOND-TERM  ATTRITION  LOSSES  BY  TERM  YEAR 

Airmen  Loss  Rates  (%) 
at  - 


FY 

Term  Year 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

7117 

1.98 

2.61 

-45 

-31.8 

-3.82 

1 

2885 

1.16 

2.36 

-35 

-103.4 

-1.2 

2 

2025 

2.48 

2.90 

-19.4 

-0.5 

-1.39 

3 

1790 

2.66 

2.29 

7 

13.9 

84 

All 

3.92 

-161 

-89.4 

-1.9 

-12.12 

1 

1.32 

3.81 

-77 

-188.6 

-2.5 

-12.16 

2 

2785 

2.44 

4.38 

-54 

-79.5 

-1.9 

-6.64 

3 

1950 

2.73 

3.54 

-16 

-29.7 

-2.19 

85 

All 

2.26 

4.86 

-242 

-115.0 

-2.6 

-16.88 

1 

2513 

1.33 

4.54 

-81 

-241.4 

-3.2 

-14.05 

2 

3258 

2.59 

-79 

-94.2 

-2.4 

-8.77 

3 

2639 

2.68 

5.27 

-68 

-96.6 

-2.6 

-8.24 

’Carter  et  al.  (1987).  Table  3.1. 
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FY  85,  while  the  model’s  estimates  changed  very  little.  As  a  consequence,  the  errors 
were  substantial  by  any  measure  in  FY  85.  The  discrepancy  is  consistent  over  all  the 
subgroups  analyzed,  but  is  not  so  severe  for  higher  term  years,  presence  of  bonus,  grade 
E-5  rather  than  grade  E-4,  and  six-year  term  of  enlistment.  These  observations  suggest 
that  older,  more  experienced  airmen  may  be  appropriately  modeled,  but  that  younger 
airmen  are  not.  Recalling  that  the  number  of  airmen  to  which  this  model  has  been 
applied  has  increased  considerably  between  FY  83  and  FY  84  suggests  the  possibility  of 
an  influx  of  a  certain  group  of  airmen  that  ought  to  be  modeled  differently.  Refitting  with 
more  recent  data  should  take  care  of  the  problem. 

Second-Term  ETS 

The  second-term  ETS  loss  model  has  two  parts.  One  part  calculates  loss  rates;  the 
other,  extend-given-stay  rates. 

Table  2.18 

SECOND-TERM  ATTRITION  LOSSES  BY  CAREER  FIELD  GROUP 


Table  2.19 


SECOND-TERM  ATTRITION  LOSSES  BY  BONUS 


FY 

Bonus 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

7117 

1.98 

2.61 

-45 

-31.8 

-0.6 

-3.82 

None 

4198 

2.19 

2.26 

-3 

-3.2 

-0.31 

1 

2201 

1.74 

2.42 

-15 

-39.1 

-2.44 

84 

All 

3.92 

-161 

-89.4 

-1.9 

-12.12 

None 

5584 

2.19 

-83.1 

-1.8 

-9.29 

1 

2263 

1.83 

3.14 

-71.6 

-1.3 

-4.65 

85 

All 

9308 

2.26 

4.86 

-242 

mm 

-2.6 

-16.88 

None 

5749 

2.44 

5.27 

-163 

-2.8 

-13.91 

1 

1881 

2.34 

4.63 

-43 

-97.9 

-2.3 

-6.57 

Table  2.20 

SECOND-TERM  ATTRITION  LOSSES  BY  GRADE 


Airmen  Loss  Rates  (%) 
at 

FY  Grade  Risk  Predicted  Actual  PE  PRE  (%)  PEPNR  (%)  SPE 


-27- 


a 


Table  2.21 

SECOND-TERM  ATTRITION  LOSSES  BY  TERM  OF  ENLISTMENT 


FY 

TOE 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

7117 

1.98 

2.61 

^15 

-31.8 

-0.6 

-3.82 

4 

4639 

2.03 

2.61 

-27 

-28.6 

-0.6 

-2.80 

6 

2294 

1.97 

2.66 

-16 

-35.0 

-0.7 

-2.38 

84 

All 

8702 

2.07 

3.92 

-161 

-89.4 

-1.9 

-12.12 

4 

5126 

2.11 

4.31 

-113 

-KM. 3 

-2.2 

-10.96 

6 

3320 

2.07 

3.31 

-41 

-59.9 

-1.2 

-5.02 

85 

All 

9308 

2.26 

4.86 

-242 

-115.0 

-2.6 

-16.88 

4 

5212 

2.26 

5.83 

-186 

-158.0 

-3.6 

-17.34 

6 

3939 

2.35 

3.67 

-52 

-56.2 

-1.3 

-5.47 

Loss  Rates 

The  second-term  ETS  model10  applied  to  some  1,200-1,400  airmen  in  the  three 
fiscal  years  studied,  or  about  3  percent  of  the  sample.  The  performance  of  this  model  is 
examined  with  respect  to  CFG  (Table  2.22),  bonus  (Table  2.23),  and  grade  (Table  2.24). 

Generally,  the  model  performance  is  fair  in  FY  83  and  FY  84,  and  a  little  worse 
but  not  terrible  in  FY  85.  It  tracks  differences  in  CFGs  and  grade  fairly  well.  Actual  loss 
rates  increased  sharply  in  FY  85,  and  the  predicted  loss  rates  followed,  although  the  gap 
between  predicted  and  actual  loss  rates  widened.  It  is  comforting  that  the  model  at  least 
changed,  and  if  the  reasons  for  the  general  ovcrprediction  seen  in  FY  83  could  be 
diagnosed  and  repaired,  the  performance  in  FY  85  would  no  doubt  improve  substantially. 
In  FY  85  there  is  a  sharp  decrease  in  loss  rates  for  airmen  offered  a  bonus  multiple  1, 
while  the  model  predicts  a  slight  increase.  Evidently  a  bonus  multiple  of  1  had  little 
effect  in  the  years  used  to  estimate,  but  something  changed  in  FY  85.  Loss  rates  for 
airmen  at  grade  5  arc  somewhat  lower  than  for  airmen  at  grade  4,  and  the  model  does  a 
good  job  of  capturing  this  effect. 

10Carter  ct  al.  (1987),  Table  6.1. 
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SECOND-TERM  ETS  LOSSES  BY  GRADE 


FY 

Grade 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

1203 

14.26 

12.05 

27 

15.5 

2.2 

2.19 

E-4 

223 

14.35 

13 

29.2 

5.9 

2.20 

E-5 

938 

13.16 

11.62 

14 

11.7 

1.5 

1.40 

84 

All 

15.72 

13.26 

35 

15.6 

2.5 

2.54 

E-4 

256 

22.28 

19.53 

7 

12.3 

2.7 

E-5 

1074 

14.70 

11.45 

35 

22.1 

3.2 

85 

All 

1390 

21.88 

17.27 

64 

21.1 

4.6 

4.16 

E-4 

282 

28.62 

21.99 

19 

23.2 

6.6 

2.46 

E-5 

15.88 

48 

23.1 

4.8 

3.75 

Extend-Given-Stay  Rates 

The  second-term  ETS  extend-given-stay  model11  applies  to  about  1,300  airmen, 
with  a  loss  rate  of  around  12  percent,  hence  approximately  1,100  airmen  remain  and  are 
faced  with  the  extend-given-stay  decision.  The  performance  of  this  model  is  examined 
with  respect  to  Career  Field  Group  (Table  2.25),  bonus  (Table  2.26),  and  grade  (Table 
2.27). 

The  performance  of  the  model  is  not  too  good.  Both  actual  and  predicted  extend- 
given-stay  rates  remained  about  the  same  in  FY  83  and  FY  84,  and  jumped  sharply  in  FY 
85,  so  at  least  the  model  appears  to  be  responsive  to  change.  The  problem  lies  in  the  fact 
that  the  model  overpredicted  the  rates  in  all  three  years.  The  error  is  not  so  severe  in 
some  smaller  subgroups,  but  because  the  sample  sizes  are  small  it  is  not  certain  whether 
the  superior  performance  is  due  to  chance. 

Second-Term  Extension 

The  second-term  extension  model12  applies  to  just  over  1,000  airmen  in  FY  83, 
and  even  fewer  in  FY  84  and  FY  85.  The  performance  of  this  model  is  examined  with 
respect  to  Career  Field  Group  (Table  2.28)  and  decisionmaker/nondccisionmaker  status 
(Table  2.29). 

1  ‘Carter  et  al.  (1987),  Table  6.2. 

I2Carteretal.  (1987),  Table  8.1  and  Table  8.2. 
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Tabic  2.25 


SECOND-TERM  EXTEND-GIVEN-STAY  BY  CAREER  FIELD  GROUP 


FY 

Career 
Field  Group 

Airmen 

at 

Risk 

Stay  Rates 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

1058 

40.94 

36.11 

51 

11.8 

4.2 

3.00 

SkilTcch 

358 

42.94 

34.92 

29 

18.7 

6.6 

2.78 

ElMcRcpa 

280 

35.94 

28.57 

21 

20.5 

6.8 

2.46 

Support 

281 

42.84 

41.28 

4 

3.6 

1.4 

0.50 

CrfScrSu 

136 

41.07 

44.85 

-5 

-9.2 

-3.5 

-0.86 

84 

All 

1223 

41.24 

36.39 

59 

11.8 

4.2 

3.21 

SkilTcch 

340 

44.24 

32.65 

39 

26.2 

9.3 

3.86 

ElMcRcpa 

322 

35.91 

32.61 

11 

9.2 

3.0 

1.18 

Support 

370 

41.93 

40.81 

4 

2.7 

1.0 

0.41 

CrfSerSu 

183 

42.60 

40.98 

3 

3.8 

1.4 

0.42 

85 

All 

1150 

49.75 

41.22 

98 

17.1 

7.1 

5.26 

SkilTcch 

392 

52.67 

41.84 

42 

20.6 

8.8 

3.88 

ElMcRcpa 

300 

45.23 

34.33 

33 

24.1 

9.0 

3.45 

Support 

303 

50.54 

50.50 

0 

0.1 

0.0 

0.01 

CrfSerSu 

149 

49.13 

33.56 

23 

31.7 

13.0 

3.48 

Tabic  2.26 


SECOND-TERM  EXTEND-GIVEN-STAY  BY  BONUS 


FY 

Bonus 

Airmen 

at 

Risk 

Stay  Rates 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

1058 

40.94 

36.11 

51 

11.8 

4.2 

3.00 

None 

701 

45.08 

41.23 

27 

8.5 

3.4 

1.93 

1 

279 

35.94 

25.09 

30 

30.2 

9.5 

3.54 

84 

All 

1223 

41.24 

36.39 

59 

11.8 

4.2 

3.21 

None 

947 

43.62 

38.75 

46 

11.2 

4.2 

2.82 

1 

233 

35.04 

29.18 

14 

16.7 

5.1 

1.75 

85 

All 

1150 

49.75 

41.22 

98 

17.1 

7.1 

5.26 

None 

892 

51.84 

43.83 

71 

15.5 

6.6 

4.36 

1 

157 

43.31 

42.86 

1 

1.0 

0.3 

0.10 

vv  v.  v_  w  i v. 
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Table  2.27 


SECOND-TERM  EXTEND-G1VEN-STAY  BY  GRADE 


FY 

Grade 

Airmen 

at 

Risk 

Stay  Rates 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted 

Actual 

83 

All 

1058 

40.94 

36.11 

51 

11.8 

4.2 

3.00 

E-4 

191 

45.87 

42.93 

6 

6.4 

2.5 

0.75 

E-5 

561 

34.62 

36.30 

-9 

-4.9 

-1.0 

-0.65 

84 

All 

1223 

41.24 

36.39 

59 

11.8 

4.2 

3.21 

E-4 

206 

47.25 

44.66 

5 

5.5 

2.1 

0.67 

E-5 

951 

40.37 

34.28 

58 

15.1 

5.4 

3.60 

85 

All 

1150 

49.75 

41.22 

98 

17.1 

7.1 

5.26 

E-4 

220 

54.79 

48.18 

15 

12.1 

5.2 

1.74 

E-5 

853 

48.84 

38.69 

87 

20.8 

8.5 

5.44 

Table  2.28 

SECOND-TERM  EXTENSION  LOSSES  BY  CAREER  HELD  GROUP 


FY 

Career 
Field  Group 

Airmen 

at 

Risk 

Loss  Rates  (%) 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted 

Actual 

83 

All 

1037 

14.71 

14.27 

5 

3.0 

0.4 

0.40 

SkilTech 

409 

16.47 

14.43 

8 

12.4 

2.0 

1.11 

ElMeRcpa 

246 

14.51 

15.45 

-2 

-6.5 

-0.9 

-0.42 

Support 

254 

11.84 

12.20 

-1 

-3.0 

-0.4 

-0.18 

CrfSerSu 

118 

14.94 

16.10 

-1 

-7.8 

-1.2 

-0.35 

84 

All 

759 

14.92 

14.10 

6 

5.5 

0.8 

0.63 

SkilTech 

272 

17.73 

18.75 

-3 

-5.8 

-1.0 

-0.44 

ElMeRepa 

157 

13.26 

12.74 

1 

3.9 

0.5 

0.19 

Support 

230 

13.55 

13.48 

0 

0.5 

0.1 

0.03 

CrfSerSu 

97 

13.45 

5.15 

8 

61.7 

8.3 

2.40 

85 

All 

736 

19.09 

14.67 

33 

23.2 

4.4 

3.05 

SkilTech 

225 

21.88 

18.22 

8 

16.7 

3.7 

1.33 

ElMeRepa 

158 

19.13 

11.39 

12 

40.5 

7.7 

2.47 

Support 

234 

16.47 

11.97 

11 

27.3 

4.5 

1.86 

CrfSerSu 

117 

18.45 

17.95 

1 

2.7 

0.5 

0.14 

The  model  does  a  very  good  job  in  FY  83,  with  a  standardized  prediction  error 
well  within  the  expected  range.  It  does  nearly  as  well  in  FY  84,  but  in  FY  85  a  sharp 
increase  in  predicted  loss  rates  is  not  matched  by  what  really  happened.  Unlike  first-term 
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extenders,  the  fraction  in  the  "decisionmaker"  category  remained  about  the  same 
between  FY  83  and  FY  85.  There  was  a  slight  increase  in  the  loss  rate  among 
decisionmakers  in  FY  85,  and  a  corresponding,  but  much  larger,  increase  in  the  predicted 
loss  rate.  The  error  seen  in  FY  85  can  be  attributed  largely  to  this  overprediction. 

Career  Attrition 

The  career  attrition  model13  applies  to  between  9,000  and  10,000  airmen,  or  about 
20  percent  of  the  sample.  The  performance  of  the  model  is  examined  with  respect  to 
term  year  (Table  2.30),  Career  Field  Group  (Table  2.31),  bonus  (Table  2.32),  grade 
(Table  2.33),  and  term  of  enlistment  (Table  2.34). 

This  model  substantially  underpredicts  the  actual  loss  rates.  It  does  so  in  FY  83,  a 
year  used  to  estimate  it,  and  by  an  even  greater  fraction  in  FY  85.  Actual  loss  rates  are 
low,  however,  so  when  the  error  is  calculated  as  a  percent  of  the  number  of  airmen  at 
risk,  this  error  is  very  low.  Thus,  the  indifferent  performance  of  this  model  may  not 
affect  estimates  of  total  losses  when  they  are  combined  with  those  of  other  models. 


Table  2.29 

SECOND-TERM  EXTENSION  LOSSES  BY  ATTRITION/DECISIONMAKER 


Airmen  Loss  Rates  (%) 

Attrition  at 

FY  /Decision  Risk  Predicted  Actual  PE  PRE  (%)  PEPNR  (%)  SPE 


CAREER  ATTRITION  LOSSES  BY  TERM  YEAR 


FY 

Term  Year 

Airmen 

at 

Risk 

Loss  Rates  (%) 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted 

Actual 

83 

All 

9169 

0.68 

1.06 

-35 

-55.9 

-0.4 

-4.43 

1 

3167 

0.46 

0.85 

-12 

-84.8 

-0.4 

-3.24 

2 

2908 

0.72 

1.10 

-11 

-52.8 

-0.4 

-2.42 

3 

2507 

0.85 

1.08 

-6 

-27.1 

-0.2 

-1.25 

84 

All 

9631 

0.67 

1.09 

^10 

-62.7 

-0.4 

-5.05 

1 

3243 

0.46 

0.74 

-9 

-60.9 

-0.3 

-2.36 

2 

3025 

0.70 

1.32 

-19 

-88.6 

-0.6 

-4.09 

3 

2733 

0.83 

1.28 

-12 

-54.2 

-0.5 

-2.59 

85 

All 

9910 

0.70 

1.49 

-78 

-112.9 

-0.8 

-9.43 

1 

2852 

0.43 

0.98 

-16 

-127.9 

-0.6 

-4.49 

2 

3115 

0.73 

2.02 

-40 

-176.7 

-1.3 

-8.46 

3 

2843 

0.83 

1.55 

-20 

-86.7 

-0.7 

-4.23 

Table  2.31 

CAREER  ATTRITION  LOSSES  BY  CAREER  HELD  GROUP 

FY 

Career 
Field  Group 

Airmen 

at 

Risk 

Loss  Rates  (%) 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted  Actual 

83 

All 

9169 

0.68 

1.06 

-35 

-55.9 

-0.4 

-4.43 

SkilTech 

2857 

0.62 

0.95 

-9 

-53.2 

-0.3 

-2.25 

ElMeRepa 

2868 

0.54 

0.84 

-9 

-55.6 

-0.3 

-2.19 

Support 

2303 

0.80 

1.13 

-8 

-41.2 

-0.3 

-1.78 

CrfSerSu 

1089 

0.93 

1.65 

-8 

-77.4 

-0.7 

-2.48 

84 

All 

9631 

0.67 

1.09 

-40 

-62.7 

-0.4 

-5.05 

SkilTech 

3081 

0.63 

1.30 

-21 

-106.3 

-0.7 

-4.70 

ElMeRepa 

2963 

0.54 

0.71 

-5 

-31.5 

-0.2 

-1.26 

Support 

2416 

0.77 

1.41 

-15 

-83.1 

-0.6 

-3.60 

CrfSerSu 

1126 

0.91 

0.89 

0 

2.2 

0.0 

0.07 

85 

All 

9910 

0.70 

1.49 

-78 

-112.9 

-0.8 

-9.43 

SkilTech 

3085 

0.67 

1.17 

-15 

-74.6 

-0.5 

-3.40 

ElMeRepa 

3098 

0.58 

1.32 

-23 

-127.6 

-0.7 

-5.42 

Support 

2420 

0.78 

1.78 

-24 

-128.2 

-1.0 

-5.59 

CrfSerSu 

1268 

0.92 

1.89 

-12 

-105.4 

-1.0 

-3.62 

Table  2.32 


CAREER  ATTRITION  LOSSES  BY  BONUS 


Airmen  Loss  Rates  (%) 
at 


FY 

Bonus 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR 

83 

All 

9169 

0.68 

1.06 

-35 

-55.9 

-0.4 

None 

8520 

0.66 

0.93 

-23 

-40.9 

-0.3 

1 

540 

0.87 

1.67 

-92.0 

-0.8 

84 

All 

9631 

0.67 

1.09 

-40 

-62.7 

-0.4 

None 

8975 

0.66 

0.94 

-25 

—42.4 

-0.3 

1 

520 

0.81 

2.31 

-8 

-185.2 

-1.5 

85 

All 

9910 

0.70 

1.49 

-78 

-112.9 

-0.8 

None 

8556 

0.71 

1.48 

-66 

-108.5 

-0.8 

1 

408 

1.17 

2.45 

-5 

-109.4 

-1.3 
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Table  2.33 


CAREER  ATTRITION  LOSSES  BY  GRADE 


FY 

Grade 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%)  PEPNR 

83 

All 

9169 

0.68 

1.06 

-35 

-55.9 

-0.4 

E-5 

4233 

0.92 

1.20 

-12 

-30.4 

-0.3 

E-6 

3355 

0.39 

0.60 

-7 

-53.8 

-0.2 

E-7 

1203 

0.34 

0.08 

3 

76.5 

0.3 

84 

All 

9631 

0.67 

1.09 

-40 

-62.7 

-0.4 

E-5 

4360 

0.90 

1.12 

-10 

-24.4 

-0.2 

E-6 

3511 

0.39 

0.43 

-1 

-10.3 

-0.0 

E-7 

1334 

0.34 

0.37 

—0 

-8.8 

-0.0 

85 

All 

9910 

0.70 

1.49 

-78 

-112.9 

-0.8 

E-5 

4279 

0.98 

1.73 

-32 

-76.5 

-0.8 

E-6 

3697 

0.42 

0.41 

0 

2.4 

0.0 

E-7 

1477 

0.35 

0.20 

2 

42.9 

0.1 

m 

m 

% 


sw 

m 


CAREER  ATTRITION  LOOSES  BY  TERM  OF  ENLISTMENT 


FY 

TOE 

Airmen 

at 

Risk 

Loss  Rates  {%) 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted 

Actual 

83 

All 

9169 

0.68 

1.06 

-35 

-55.9 

-0.4 

-4.43 

4 

5562 

0.72 

1.10 

-21 

-52.8 

-0.4 

-3.35 

6 

2476 

0.67 

1.13 

-11 

-68.7 

-0.5 

-2.81 

84 

All 

9631 

0.67 

1.09 

-40 

-62.7 

-0.4 

-5.05 

4 

5095 

0.39 

0.90 

-26 

-130.8 

-0.5 

-5.84 

6 

3310 

0.70 

1.42 

-24 

-102.9 

-0.7 

-4.97 

85 

All 

9910 

0.70 

1.49 

-78 

-112.9 

-0.8 

-9.43 

4 

4814 

0.69 

1.58 

-43 

-129.0 

-0.9 

-7.46 

6 

3843 

0.78 

1.29 

-20 

-65.4 

-0.5 

-3.59 

Career  ETS 

The  career  ETS  model  has  two  parts.  One  part  calculates  loss  rates;  the  other, 
extend-given-stay  rates. 

Loss  Rates.  The  career  ETS  model14  applies  to  about  1,600  airmen  in  the  sample, 
about  3  percent  of  the  entire  sample.  Its  performance  is  examined  with  respect  to  Career 
Field  Group  (Table  2.35)  and  grade  (Table  2.36). 

Generally,  the  model  does  quite  well  in  each  of  the  three  fiscal  years  and  among 
the  strata  considered  here.  Even  in  FY  85,  two  years  beyond  the  last  year  used  to 
estimate  the  model,  the  standardized  prediction  error  remains  well  within  acceptable 
limits. 

The  tables  show  rather  high  relative  errors  because  the  actual  number  of  losses  is 
extremely  low  (sometimes  only  five  or  six  airmen).  This  is  no  cause  for  concern, 
because  the  other  error  measures,  better  indicators  of  validity,  show  the  model  is 
performing  accurately. 

Extend-Given-Stay  Rates.  The  career  ETS  model  applies  to  around  1 ,600 
airmen,  with  a  loss  rate  of  less  than  2  percent.  Hence,  approximately  1,600  airmen  are 
faced  with  the  "extend-given-stay"  decision.  The  performance  of  this  model15  is 
examined  with  respect  to  Career  Field  Group  (Table  2.37)  and  grade  (Table  2.38). 


14Carter  et  al.  (1987),  Table  7.5. 
15Carter  et  al.  (1987),  Table  7.1. 


Table  2.35 


CAREER  ETS  LOSSES  BY  CAREER  FIELD  GROUP 


FY 

Career 
Field  Group 

Airmen 

at 

Risk 

Loss  Rates  (%) 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted 

Actual 

83 

All 

1617 

1.76 

1.67 

1 

5.1 

0.1 

0.28 

SkilTech 

501 

2.30 

1.80 

3 

21.7 

0.5 

C.75 

ElMeRepa 

450 

1.52 

0.67 

4 

55.9 

0.8 

1.47 

Support 

466 

1.48 

1.93 

-2 

-30.4 

-0.5 

-0.80 

CrfSerSu 

190 

1.65 

3.16 

-3 

-91.5 

-1.5 

-1.63 

84 

All 

1741 

2.16 

2.35 

-3 

-8.8 

-0.2 

-0.55 

SkilTech 

476 

2.85 

2.52 

2 

11.6 

0.3 

0.43 

ElMeRepa 

532 

1.74 

2.26 

-3 

-29.9 

-0.5 

-0.92 

Support 

481 

1.80 

1.46 

2 

18.9 

0.3 

0.56 

CrfSerSu 

246 

1.91 

4.07 

-5 

-113.1 

-2.2 

-2.48 

85 

All 

1639 

2.55 

2.93 

-6 

-14.9 

-0.4 

-0.98 

SkilTech 

465 

2.91 

1.94 

5 

33.3 

1.0 

1.24 

ElMeRepa 

474 

2.44 

3.59 

-5 

-47.1 

-1.2 

-1.62 

Support 

488 

2.31 

3.07 

-4 

-32.9 

-0.8 

-1.12 

CrfSerSu 

201 

2.56 

2.99 

-1 

-16.8 

-0.4 

-0.39 

Table  2.36 

CAREER  ETS  LOSSES  BY  GRADE 


FY 

Grade 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

1617 

1.76 

1.67 

1 

5.1 

0.1 

0.28 

E-5 

601 

3.62 

3.33 

2 

8.0 

0.3 

0.38 

E-6 

760 

0.75 

0.53 

2 

29.3 

0.2 

0.70 

E-7 

231 

0.32 

0.87 

-1 

-171.9 

-0.6 

-1.48 

84 

All 

1741 

2.16 

2.35 

-3 

-8.8 

-0.2 

-0.55 

E-5 

600 

4.19 

4.00 

1 

4.5 

0.2 

0.23 

E-6 

859 

0.98 

1.05 

-1 

-7.1 

-0.1 

-0.21 

E-7 

250 

0.38 

0.80 

-1 

-110.5 

-0.4 

-1.08 

85 

All 

1639 

2.55 

2.93 

-6 

-14.9 

-0.4 

-0.98 

E-5 

567 

4.82 

4.59 

1 

4.8 

0.2 

0.26 

E-6 

751 

1.41 

1.33 

1 

5.7 

0.1 

0.19 

E-7 

271 

0.52 

0.37 

0 

28.8 

0.2 

0.34 
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Table  2.37 


CAREER  EXTEND-GIVEN-STAY  BY  CAREER  FIELD  GROUP 


FY 

Career 
Field  Group 

Airmen 

at 

Risk 

Stay  Rates 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

1590 

22.50 

28.99 

-103 

-28.8 

-6.4 

-6.15 

SkilTech 

492 

22.51 

29.47 

-34 

-30.9 

-6.8 

-3.66 

ElMeRepa 

447 

21.61 

28.64 

-31 

-32.5 

-7.0 

-3.60 

Support 

457 

23.14 

31.07 

-36 

-34.3 

-7.8 

-3.98 

CrfSerSu 

184 

23.18 

22.83 

1 

1.5 

0.3 

0.11 

84 

All 

1700 

21.93 

28.53 

-112 

-30.1 

-6.4 

-6.50 

SkilTech 

464 

22.61 

30.17 

-35 

-33.4 

-7.4 

-3.84 

ElMeRepa 

520 

21.64 

24.23 

-13 

-12.0 

-2.5 

-1.42 

Support 

474 

22.05 

33.97 

-56 

-54.1 

-11.7 

-6.21 

CrfSerSu 

236 

20.91 

23.31 

-6 

-11.5 

-2.3 

-0.89 

85 

All 

1591 

21.70 

29.16 

-119 

-34.4 

-7.2 

-7.11 

SkilTech 

465 

22.13 

29.82 

-36 

-34.7 

-7.5 

-3.95 

ElMeRepa 

457 

21.86 

27.27 

-25 

-24.7 

-5.2 

-2.75 

Support 

473 

20.87 

28.12 

-34 

-34.7 

-7.0 

-3.82 

CrfSerSu 

195 

22.15 

33.33 

-22 

-50.5 

-10.8 

-3.70 

Table  2.38 


CAREER  EXTEND-GIVEN-STAY  BY  GRADE 


FY 

Grade 

Airmen 

at 

Risk 

Stay  Rates 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

1590 

22.50 

28.99 

-103 

-28.8 

-6.4 

-6.15 

E-5 

581 

23.32 

29.09 

-34 

-24.7 

-5.6 

-3.23 

E-6 

756 

21.74 

28.70 

-53 

-32.0 

-6.9 

-4.63 

84 

All 

1700 

21.93 

28.53 

-112 

-30.1 

-6.4 

-6.50 

E-5 

576 

22.52 

30.21 

—44 

-34.1 

-7.4 

-4.33 

E-6 

850 

21.10 

27.65 

-56 

-31.0 

-6.5 

-4.66 

85 

All 

1591 

21.70 

29.16 

-119 

-34.4 

-7.2 

-7.11 

E-5 

541 

22.47 

30.50 

-43 

-35.7 

-7.7 

—4.37 

E-6 

741 

20.77 

29.96 

-68 

-44.2 

-9.1 

-6.13 
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Both  predicted  and  actual  rates  remained  about  the  same  from  one  year  to  the 
next,  with  predictions  falling  substantially  below  the  actual  rates.  The  underprediction  is 
about  the  same  in  all  subgroups.  The  fact  that  the  standardized  prediction  error  is  lower 
in  some  groups  simply  reflects  the  fact  that  the  sample  size  is  smaller. 

Career  Extension 

The  career  extension  model16  applies  to  fewer  than  1,000  airmen,  less  than  2 
percent  of  the  sample.  Its  performance  is  considered  by  Career  Field  Group  (Table 
2.39),  grade  (Table  2.40),  and  the  decision/nondecision  state  (Table  2.41). 

This  model  does  well  in  each  of  the  three  fiscal  years,  and  within  most  of  the  strata 
considered.  As  in  the  career  ETS  model,  some  relative  errors  are  high  because  of  the 
small  number  of  actual  losses  represented,  and  this  is  no  cause  for  concern. 

As  in  the  previous  extension  models,  the  loss  rate  among  decisionmakers  was 
somewhat  higher  in  FY  85  than  in  FY  83.  The  career  model  matched  this  increase  very 
closely,  predicting  virtually  the  same  number  of  losses. 

Table  2.39 

CAREER  EXTENSION  LOSSES  BY  CAREER  HELD  GROUP 


Career 


Airmen  Loss  Rates 
at  - 


FY 

Field  Group 

Risk 

Predicted 

Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

829 

3.77 

4.34 

-5 

-15.1 

-0.6 

-0.86 

SkilTech 

267 

4.10 

3.37 

2 

17.8 

0.7 

0.60 

ElMeRepa 

233 

3.01 

3.86 

-2 

-28.2 

-0.9 

-0.76 

Support 

232 

3.19 

6.47 

-8 

-102.8 

-3.3 

-2.84 

CrfSerSu 

92 

6.34 

3.26 

3 

48.6 

3.1 

1.21 

84 

All 

690 

4.10 

2.90 

8 

29.3 

1.2 

1.59 

SkilTech 

218 

4.65 

3.67 

2 

21.1 

1.0 

0.69 

ElMeRepa 

185 

2.88 

1.08 

3 

62.5 

1.8 

1.46 

Support 

205 

3.60 

3.41 

0 

5.3 

0.2 

0.15 

CrfSerSu 

76 

6.89 

3.95 

2 

42.7 

2.9 

1.01 

85 

All 

679 

4.67 

5.15 

-3 

-10.3 

-0.5 

-0.59 

SkilTech 

199 

5.37 

7.54 

-4 

-40.4 

-2.2 

-1.36 

ElMeRepa 

185 

3.40 

2.16 

2 

36.5 

1.2 

0.93 

Support 

219 

4.08 

5.02 

-2 

-23.0 

-0.9 

-0.70 

CrfSerSu 

70 

8.01 

5.71 

2 

28.7 

2.3 

0.71 

I6Carter  et  al.  (1987),  Table  8.1  and  Table  8.2. 
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Table  2.40 


CAREER  EXTENSION  LOSSES  BY  GRADE 


FY 

Career 
Field  Group 

Airmen 

at 

Risk 

Loss  Rates  (%) 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted 

Actual 

83 

All 

829 

3.77 

4.34 

-5 

-15.1 

-0.6 

-0.86 

E-5 

286 

4.17 

7.69 

-10 

-84.4 

-3.5 

-2.98 

E-6 

390 

3.58 

2.31 

5 

35.5 

1.3 

1.35 

84 

All 

690 

4.10 

2.90 

8 

29.3 

1.2 

1.59 

E-5 

187 

4.39 

4.81 

-1 

-9.6 

-0.4 

-0.28 

E-6 

345 

4.04 

2.03 

7 

49.8 

2.0 

1.90 

85 

All 

679 

4.67 

5.15 

-3 

-10.3 

-0.5 

-0.59 

E-5 

198 

5.17 

8.08 

-6 

-56.3 

-2.9 

-1.85 

E-6 

320 

4.72 

4.37 

1 

7.4 

0.3 

0.30 

Table  2.41 

CAREER  EXTENSION  LOSSES  BY  ATTRITION/DECISIONMAKER 

FY 

Attrition 

/Decision 

Airmen 

at 

Risk 

Loss  Rates  (%) 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

Predicted 

Actual 

83 

All 

829 

3.77 

4.34 

-5 

-15.1 

-0.6 

-0.86 

Attrition 

451 

1.34 

1.55 

-1 

-15.7 

-0.2 

-0.39 

Decision 

378 

6.66 

7.67 

-4 

-15.2 

-1.0 

-0.79 

84 

All 

690 

4.10 

2.90 

8 

29.3 

1.2 

1.59 

Attrition 

373 

1.33 

0.27 

4 

79.7 

1.1 

1.79 

Decision 

317 

7.35 

5.99 

4 

18.5 

1.4 

0.93 

85 

All 

679 

4.67 

5.15 

-3 

-10.3 

-0.5 

-0.59 

Attrition 

406 

1.34 

2.22 

-4 

-65.7 

-0.9 

-1.54 

Decision 

273 

9.62 

9.52 

0 

1.0 

0.1 

0.06 

Retirement 

The  retirement  model17  applies  to  about  3,300  airmen  each  fiscal  year,  6  percent 
of  the  sample.  Its  performance  is  examined  with  respect  to  Career  Field  Group  (Table 
2.42),  grade  (Table  2.43),  and  term  year  (Table  2.44). 

This  model’s  performance  is  quite  good.  Overall,  it  docs  better  in  FY  84  and  FY 
85  than  it  did  in  FY  83.  The  standardized  prediction  error  is  usually  within  the  expected 
range. 

17Carter  et  al.  (1987),  Table  9.3. 


Table  2.43 


RETIREMENT  LOSSES  BY  GRADE 


FY 

Grade 

Airmen 

at 

Risk 

Loss  Rates  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

83 

All 

3421 

25.78 

27.27 

-51 

-5.8 

-1.5 

-1.99 

E-6 

587 

33.25 

34.07 

-5 

-2.5 

-0.8 

-0.42 

E-7 

1641 

25.82 

27.36 

-25 

-6.0 

-1.5 

-1.43 

E-8 

761 

20.24 

22.21 

-15 

-9.7 

-2  0 

-1.35 

84 

All 

3360 

24.51 

24.40 

4 

0.4 

0.1 

0.15 

E-6 

523 

33.93 

33.46 

2 

1.4 

0.5 

0.23 

E-7 

1642 

24.63 

24.67 

-1 

-0.2 

-0.0 

-0.04 

E-8 

755 

18.10 

16.16 

15 

10.7 

1.9 

1.38 

85 

All 

3315 

27.76 

28.02 

-9 

-0.9 

-0.3 

-0.33 

E-6 

511 

37.37 

35.62 

9 

4.7 

1.7 

0.82 

E-7 

1631 

27.12 

29.31 

-36 

-8.1 

-2.2 

-1.99 

E-8 

733 

21.66 

19.37 

17 

10.6 

2.3 

1.51 
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Table  2.44 

RETIREMENT  LOSSES  BY  TERM  YEAR 

Airmen  Loss  Rates  (%) 
at 

FY  Term  Year  Risk  Predicted  Actual  PE  PRE  (%)  PEPNR  (%)  SPE 
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III.  FISCAL  YEAR  TEST  AND  EVALUATION 


INTRODUCTION 


This  section  discusses  the  fiscal  year  test  and  evaluation  of  the  middle-term 
disaggregate  loss  models.  These  models,  described  by  Carter  et  al.  (1987),  estimate  loss 
rates  for  an  airman’ '  ’  car  at  risk.1  Such  loss  rates  by  themselves  are  not  particularly 
useful.  Planners  nee^.  1  predict  loss  rates  for  a  fiscal  year,  and  a  fiscal  year  coincides 
with  the  year  at  risk  for  only  a  small  fraction  of  the  Air  Force.  The  process  of  combining 
the  disaggregate  loss  models  to  obtain  fiscal  year  loss  rates  is  called  blending ? 

Once  the  blending  process  was  formulated  and  coded,  the  combined  models  were 
used  to  estimate  loss  rates  over  a  fiscal  year  and  these  rates  were  compared  with  the  ones 
actually  observed.  This  comparison  is  called  tine  fiscal  year  evaluation. 


BLENDING 


Assume  that  we  wish  to  produce  blended  loss  rates  for  fiscal  year  1984  (FY  84). 
Of  course,  the  discussion  applies  generally  to  any  other  fiscal  year  with  obvious  changes. 
This  fiscal  year  test  and  evaluation  covers  only  FY  84,  because  that  was  the  only  fiscal 
year  when  data  not  used  for  estimation  were  available  at  the  time  of  the  evaluation. 

Consider  the  following  example:  At  the  beginning  of  FY  84  an  airman  is  selected 
and  found  to  be  five  months  into  the  ETS  year  of  his  first  term.  To  calculate  the 
probability  that  he  will  be  lost  sometime  during  FY  84,  it  is  a  bit  easier  to  think  about  the 
probability  he  will  survive  the  fiscal  year.  What  is  the  probability  he  will  still  be  in  the 
service  at  the  end  of  FY  84?  To  calculate  this  survival  probability,  it  is  necessary  to 


estimate: 


1.  The  probability  he  will  survive  for  the  remaining  seven  months  of  his 
original  ETS  year. 


!See  Sec.  II  for  a  definition  of  year  at  risk. 

2  A  different  approach  to  blending  is  used  here  from  the  approach  being  used  to 
prepare  fiscal  year  loss  and  extension  rates  as  input  to  the  EFMS’s  Middle-Term 
Disaggregate  IPM. 
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2.  The  probability  he  will  reenlist,  and  the  probability  he  will  survive  the  first 
five  months  of  his  next  term  if  he  does  reenlist. 

3.  The  probability  he  will  extend,  and  the  probability  he  will  survive  the  first 
five  .nonths  of  his  extension  if  he  does  extend. 

Blending  has  two  aspects  that  need  to  be  addressed: 

1.  More  than  one  year  at  risk  and  more  than  one  loss  model  for  each  airman 
need  to  be  considered. 

2.  Loss  rates  must  be  calculated  for  parts  of  a  year  at  risk. 

Multiple  Years  at  Risk  and  Loss  Models 

The  year  at  risk,  that  applies  to  an  airman  when  he  is  selected  at  the  beginning  of 
FY  84  is  his  beginning  year  at  risk.  The  cohort  loss  model  that  applies  to  him  at  that 
point  is  his  beginning  model. 

During  the  remainder  of  his  beginning  year  at  risk,  an  airman  may  leave  the 
service.  If  he  does  not,  then  depending  on  his  beginning  status  (ETS,  extension,  etc.),  he 
will  either  reenlist,  extend,  or  simply  remain  in  his  current  term.  Which  of  these 
"choices"  is  available  depends  on  the  beginning  model.  If  the  beginning  model  is  an  ETS 
model,  then  the  airman  can  reenlist  or  extend,  but  he  cannot  remain.  If  the  beginning 
model  is  an  attrition  model,  then  he  cannot  extend,  he  can  only  remain.3  Thus,  extending 
and  remaining  are  disjoint  choices — an  airman  never  faces  both  of  them  at  one  time.  The 
definition  of  "extending"  will  therefore  be  broadened  to  mean  "remaining."  An  airman 
who  begins  in  an  attrition  model  and  remains  in  the  service  for  another  year  will  be  said 
to  have  extended. 

As  the  beginning  year  at  risk  covered  the  beginning  of  the  fiscal  year,  so  the 
ending  year  at  risk  covers  the  end.  Two  models  are  needed  for  the  ending  year  at  risk. 

An  airman  who  does  not  leave  faces  two  choices — to  reenlist  or  to  extend.4  If  he  chooses 
to  reenlist,  the  loss  model  used  to  calculate  the  loss  probability  for  the  ending  year  at  risk 

3In  the  present  implementation  of  the  models,  he  cannot  reenlist  either,  although  in 
fact  it  is  sometimes  possible  to  do  so. 

4Although  airmen  on  attrition  status  cannot  reenlist,  reenlistment  will  be  regarded  here 
as  a  choice  that  can  be  made  with  probability  zero.  A  future  implementation  of  these 
models  can  easily  change  this  assumption. 


will  be  denoted  the  reenlisting  model.  If  he  chooses  to  extend  (and  sometimes  that  is  the 
only  choice  he  can  make),  the  corresponding  loss  model  will  be  called  the  extending 
model.  Collectively  these  two  models  are  the  ending  models.  Figure  2  displays  the 
relationship  between  beginning,  extending,  and  reenlisting  years  at  risk.  A  "triple" 
airman  record  helps  address  this  complicated  situation. 

Loss  Rates  for  Parts  of  Years 

Whatever  the  beginning  model  is,  it  will  rarely  coincide  with  the  fiscal  year.  As 
illustrated  in  Fig.  3,  it  will  generally  be  necessary  to  calculate  a  loss  rate  for  some  risk 
interval,  the  beginning  risk  interval,  that  is  less  than  a  year  and  will  fall  at  the  end  of 
some  year  at  risk.  Similarly,  whatever  the  ending  models  are,  they  will  never  coincide 
with  the  fiscal  year,  and  it  will  be  necessary  to  calculate  loss  rates  over  some  interval  of 
time  less  than  a  year,  the  ending  risk  interval.  It  is  illustrated  in  Fig.  4. 


FY  83 


FY  84 


FY  85 


Beginning 
Year  at  Risk 


Extending 
Year  at  Risk 


- ► 

• 

Reenlisting 
Year  at  Risk 

- ► 


Fig.  2 — Relationship  Between  Fiscal  Years  and  Cohort  Years 


Fiscal  Year 


Beginning 
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Beginning  Year  at  Risk 


Fig.  3 — Illustration  of  Beginning  Risk  Interval 


Fiscal  Year 
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•  • 
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Ending  Year  at  Risk 


Fig.  4 — Illustration  of  Ending  Risk  Interval 


These  data  suggest  that  for  any  year  at  risk,  the  beginning  of  the  next  fiscal  year  is 
just  as  likely  to  occur  in  the  first  month  of  the  year  at  risk  as  in  any  other  month.  Thus, 
the  event  "fiscal  year  begins"  is  uniformly  distributed  over  the  months  of  any  year  at  risk. 
The  same  cannot  be  said  of  losses,  reenlistments,  or  extensions. 

In  ETS  years,  losses  arc  heavily,  but  not  exclusively,  concentrated  at  the  end  of 
the  year  at  risk,  while  reenlistments  are  more  evenly  spread.  In  the  first  term 
reenlistments  appear  to  be  more  likely  in  the  early  part  of  the  year  than  later,  at  least  in 
FY  84.  In  the  YAR  file,  ETS  extensions  always  occur  at  the  end  of  a  year  at  risk  (by 
definition).  In  attrition  years,  losses  are  spread  somewhat  smoothly  over  the  year  but  in 
the  first  year  of  the  first  term  are  more  frequent  in  the  early  part  than  later.  The  temporal 
distribution  of  losses  for  extension  terms  is  the  most  complicated  of  all,  and  not  well 
understood.  For  example,  losses  for  airmen  in  the  second  year  of  a  two-year  extension 
are  somewhat  concentrated  at  the  end  of  that  year,  but  many  occur  earlier,  bunching  up 
at  three  months,  six  months,  and  nine  months  beyond  the  beginning  of  the  year  at  risk. 
Reenlistments  out  of  extension  status  can  occur  anytime. 

The  temporal  nonuniformity  of  losses  and  reenlistments  over  time  poses  several 
problems  for  blending.  First,  it  causes  a  sampling  problem.  Because  losses, 
reenlistments,  and  extensions  are  distributed  over  time  in  different  ways,  the  sample  of 
airmen  whose  year  at  risk  covers  the  beginning  of  some  fiscal  year  will  have  a  different 
composition  from  the  subsample  who  were  actually  still  in  the  service  when  that  fiscal 
year  began.  For  instance,  if  reenlistments  occur  anytime  but  losses  are  concentrated  at 
the  end  of  the  year  at  risk,  the  subsample  of  airmen  still  in  the  service  at  the  beginning  of 
the  fiscal  year  will  have  proportionally  fewer  reenlistees  than  the  full  sample  that 
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includes  the  airmen  who  left  before  the  fiscal  year  began.  Second,  the  nonuniformity 
means  it  is  necessary  to  figure  out  ways  of  adjusting  the  year-at-risk  loss  rates,  which  are 
annual  rates,  to  get  loss  rates  for  parts  of  a  year. 

The  Blending  Process5 

This  section  describes  the  computational  process  used  to  calculate  blended,  fiscal 
year  loss  probabilities  for  use  in  the  test  and  evaluation  of  the  middle-term  disaggregate 
loss  models. 

The  process  is  somewhat  complex,  and  comprises  five  steps: 

Step  1:  Triple.  Read  data  from  the  YAR  file,  and  create 

three  records  for  every  one  that  covers  the  beginning  of  the  fiscal 

year:  a  beginning  record,  an  extending  record,  and  a  reenlisting  record. 

Step  2:  Model.  Use  the  cohort  models  to  calculate  year-at-risk  loss 
rates  for  each  record  in  the  triple  file. 

Step  3:  Merge.  Merge  these  loss  rates  by  airman. 

Step  4:  Blend.  For  each  airman,  calculate  a  fiscal  year  loss  rate. 

Step  5:  Compare.  Read  the  YAR  file  again,  selecting  records  for  the 
next  fiscal  year,  and  compare  with  predictions. 

Step  1:  Triple.  Recall  from  the  previous  section  that  in  order  to  calculate  fiscal 
year  loss  probabilities,  as  many  as  three  different  cohort  loss  models  may  be  needed. 
Linking  the  code  that  describes  the  different  loss  models  to  create  a  single  "fiscal  year 
loss  rate  model"  turned  out  to  be  inconvenient  because  the  original  code  was  written  in 
the  SAS  language,  which  is  not  useful  for  complex  programs.  Instead,  the  code  written 


to  do  the  cohort  loss  evaluation  is  retained  as  it  stands,  and  the  estimates  combined. 
Furthermore,  using  the  same  code  makes  it  unnecessary  to  write  anything  new,  leaving 
less  room  for  error,  because  the  cohort  loss  evaluation  established  that  the  code  works 
correctly. 


Section  II,  to  test  one  of  the  cohort  loss  models,  passed  a  sample  of  year-at-risk 
records  through  the  model.  Each  model  had  a  section  that  allowed  it  to  recognize  which 
records  it  was  supposed  to  use  and  which  records  it  was  supposed  to  ignore.  For 


Alternative  solutions  to  blending  problems  are  used  in  the  program  that  creates  the 
fiscal  year  loss  and  extension  rate  inputs  for  the  Middle-Term  Disaggregate  IPM. 


example,  to  test  the  first-term  ETS  model  for  FY  84,  a  10  percent  sample  of  records  from 
the  YAR  file  for  FY  84  was  passed  through  the  first-term  ETS  model.  That  model 
selected  out  the  first-term  ETS  records  and  calculated  actual  and  estimated  losses.  It 
ignored  the  other  records. 

For  fiscal-year  evaluation,  the  triple  step  creates  a  file  that  looks  (to  the  cohort  loss 
models)  just  like  a  YAR  file.  The  main  difference  is  that  for  a  given  sample  of  airmen 
(e.g.,  a  10  percent  sample  of  airmen  present  at  the  beginning  of  FY  84),  the  triple  file  has 
three  times  as  many  records  as  would  be  used  to  do  cohort  evaluation.  For  each  airman 
in  the  original  sample,  the  triple  file  has  a  beginning  record  describing  the  airman  for  his 
beginning  model,  an  extending  record  describing  him  for  his  extending  model,  and  a 
reenhsting  record  describing  him  for  his  reenlisting  model. 

Specific  decisions  made  at  the  triple  step  are: 

•  What  sampled  airmen  are  "present"  at  the  beginning  of  the  fiscal  year? 

•  What  variables  should  be  changed  for  the  extending  models,  and  how  should 
they  be  changed? 

Who’s  There?  In  cohort  evaluation,  a  record  from  the  YAR  file  was  selected  if  it 
"covered"  the  beginning  of  a  particular  fiscal  year.  However,  as  explained  in  Sec.  II,  a 
year  at  risk  lasts  a  year,  no  matter  how  long  an  airman  lasts.  So  for  a  fraction  of  selected 
records,  the  airman  may  have  already  left  the  service  or  may  have  already  reenlisted  for 
another  term  at  the  beginning  of  the  covered  fiscal  year.  Consequently,  the  triple  step 
accepts  all  those  records  from  the  YAR  file  for  which  the  airman  is  still  serving  the 
selected  term  at  the  beginning  of  the  fiscal  year. 

Airmen  facing  an  ETS  decision  may  have  left  the  service  or  rcenlisted  in  a  new 
term  before  the  beginning  of  the  covered  fiscal  year  because  of  an  "early  out"  offer.  In 
the  YAR  file,  such  airmen  will  not  be  coded  as  "present"  at  the  beginning  of  the  fiscal 
year.  However,  to  be  consistent  with  the  practice  used  to  evaluate  the  ALPS  file  (see 
Sec.  IV),  the  triple  step  accepts  such  airmen  records  for  evaluation.  (A  corresponding 
deletion  is  made  at  the  "compare  step,"  described  below.) 

Updating  Variables  for  Subsequent  Models.  In  creating  three  records  in  the 
triple  file  from  the  one  found  in  the  YAR  file,  some  data  elements  remain  the  same  in  all 
three  records,  and  some  change.  No  data  element  changes  in  the  beginning  record,  of 


course;  that  record  is  written  out  on  the  triple  file  just  as  it  is  found  on  the  YAR  file. 

Some  data  elements  never  change — those  describing  the  airman’s  race,  his  age  when  he 
joined  the  service,  etc.  Some  data  elements  are  assumed  never  to  change,  although  they 
might — the  airman’s  marital  status,  for  example.  Some  data  elements  certainly  change, 
such  as  the  number  of  years  of  service  (incremented  by  1  for  both  ending  models),  and 
the  airman’s  term  year  (incremented  for  an  extending  model,  set  to  1  for  a  reenlisting 
model).  Finally,  one  class  of  data  elements  requires  assumptions  as  to  the  changes,  such 
as  the  airman’s  grade  and  AFSC. 

Some  variables  are  changed  in  the  same  way  for  both  ending  models.  In  ending 
models,  whether  extending  or  reenlisting,  the  airman  has  spent  one  more  year,  and  one 
year  is  added  to  the  year  of  service  variable  (YYOS12)  and  the  years  in  the  military 
variable  (STAFYR).  We  add  12  months  to  two  time-in-grade  variables  (S1TG  and 
STIG).  (Implicit  in  this  addition  i«  the  assumption  of  no  change  in  grade.)  The 
beginning  of  the  year  at  risk  is  increased  by  one  year. 

Some  variables  are  changed  in  a  particular  way  for  the  extending  model.  Some  of 
the  changes  depend  on  the  beginning  model,  but  in  every  case  the  term  year  is  increased 
by  one  year. 

If  the  beginning  model  is  the  first-term  attrition  model,  the  required  records  in  the 
YAR  file  are  typically  missing  the  bonus  coefficient  and  the  AFSC  variable.  A  zero 
coefficient  was  imputed  to  the  first,  and  if  the  AFSC  was  missing,  the  code  "99000"  was 
assigned,  the  modal  value  for  second-year,  first-term  airmen,  and  whose  loss  coefficient 
is  close  to  the  average  over  all  the  loss  coefficients.  No  first-term  airman  has  a  cross¬ 
bonus  variable  WBONC  (these  are  assigned  only  to  first-term  ETS  airmen),  so  a  value  of 
zero  was  imputed  for  this  variable.  In  the  present  implementation  some  of  these 
imputations  were  not  used.  If  an  airman  begins  a  fiscal  year  in  an  attrition  year  at  risk, 
but  ends  up  that  fiscal  year  in  an  ETS  year  at  risk,  that  ETS  decision  will  not  take  place 
until  the  next  fiscal  year.  The  present  implementation  does  not  use  the  ETS  loss 
calculation  for  a  subsequent  year  at  risk,  only  for  a  beginning  year  at  risk.  Instead,  an 
attrition  loss  is  calculated  using  another  model;  see  "ending  survival  rates"  below. 

If  the  beginning  model  is  an  ETS  model,  then  the  extending  model  will  be  a  true 
extension  model,  and  it  is  necessary  to  decide  whether  the  airman  will  extend  for  one 
year  or  two  years.  No  modeling  has  been  done  on  this  decision,  so  for  the  purposes  of 
test  and  evaluation,  the  fraction  of  all  extenders  in  their  first  year  who  were  signed  up  for 


i 
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a  two-year  extension  were  calculated  and  assigned  one-  and  two-year  extensions  to 
airmen  randomly  in  proportion  to  this  fraction. 

Finally,  some  variables  were  changed  in  a  particular  way  only  for  reenlisting 
models.  All  reenlistments  were  assumed  to  be  for  four-year  terms  (by  setting 
TXTOE=4)  and  the  term  year  (YNTYR)  changed  to  1. 

Step  2:  Model.  During  the  model  step  the  triple  file  is  simply  passed  through  each 
of  the  cohort  loss  models  in  almost  exactly  the  same  way  as  for  cohort  evaluation.  The 
main  difference  is  that  different  output  files  are  created.  During  the  evaluation,  the 
models  created  tables  describing  performance;  during  blending,  the  models  create  files 
containing  estimated  loss  (and  extend-given-stay)  rates  and  an  airman  identifier,  which 
are  used  as  input  to  the  next  stage. 

Step  3:  Merge.  The  model  step  described  above  creates  10  files,  one  for  each  of 
the  10  cohort  loss  models.  Any  particular  airman  will  have  three  records  among  these  10 
files  (not  necessarily  all  different — the  beginning  and  extending  models  may  be  the  same 
for  some  airmen).  The  merge  step  simply  reassembles  the  three  records  by  airman,  so 
the  resulting  file  contains  one  record  per  airman,  and  each  record  contains  three  sets  of 
loss  rates,  one  for  the  airman’s  beginning  model,  one  for  his  extending  model,  and  one 
for  his  reenlisting  model. 

Step  4:  Blend.  What  follows  is  complicated.  For  each  airman,  the  blending  step 
uses  a  cohort  loss  rate  for  the  year  at  risk  that  covers  the  beginning  of  the  fiscal  year,  and 
two  cohort  loss  rates  for  the  year  at  risk  that  covers  the  end:  one  rate  if  the  airman 
reenlists,  another  if  he  extends.  It  also  uses  the  beginning  model’s  extend-given-stay 
rate.  It  converts  the  cohort  loss  rates  into  survival  rates  for  the  beginning  and  ending  risk 
intervals  and  then  "blends"  them  into  a  survival  rate  for  the  fiscal  year,  with  the  following 
formula: 

SURVIVE_FY  =  SURVIVE_BEG  *  (XGS*SURVIVE_EXT  + 

( 1  -XGS)*SURVIVE_REN) 

where 

SURVIVE_FY  =  Fiscal  year  survival  rate 
SURVIVE_BEG  =  Survival  rate  for  the  beginning  risk  interval 
XGS  =  Probability  of  extending  given  stay 

SURVIVE_EXT  =  Survival  rate  for  ending  risk  interval  if  airman  extends 
SURVIVE_REN  =  Survival  rate  for  ending  risk  interval  if  airman  reenlists 


Conversion  from  cohort  loss  rates  to  risk  interval  survival  rates  constitutes  the  most 
difficult  problem,  for  several  reasons. 

Solving  the  "Part-of-Year"  Problem.  For  a  given  airman  at  the  start  of  a  fiscal 
year,  the  year  at  risk  covering  that  start  usually  commenced  several  months  earlier,  so 
only  part  of  a  year  remains.  Thus,  it  is  necessary  to  calculate  the  probability  the  airman 
survives  the  last  part  of  a  year  at  risk,  given  he  survived  the  first  part. 

Similarly,  the  beginning  year  at  risk  generally  ends  several  months  before  the 
fiscal  year  ends.  Using  the  two  cohort  models  that  fill  the  gap  one  if  the  airman  reenlists, 
the  other  if  he  does  not),  it  is  necessary  to  calculate  a  survival  rate  only  for  the  first  part 
of  a  year  at  risk:  from  the  beginning  of  the  new  year  at  risk  to  the  end  of  the  old  fiscal 
year. 

The  cohort  loss  models  calculate  annual  loss  rates.  The  timing  of  these  events 
over  the  year  is  by  no  means  simple.  For  example: 

•  Losses  for  first- year  airmen  are  more  heavily  concentrated  at  the  end  of  basic 
military  training. 

•  A  substantial  fraction  of  ETS  losses  occur  before  the  end  of  the  year  at  risk. 

•  Reenlistments  can  occur  anytime  during  the  ETS  year,  but  extensions,  by 
definition,  occur  only  at  the  end. 

•  Losses  from  extension  occur  any  time,  although  a  substantial  fraction  is 
concentrated  at  the  end  of  the  extension. 

Complications  such  as  these  cause  what  a  statistician  might  call  a  sampling  bias. 

If  the  airmen  who  are  present  at  the  beginning  of  a  fiscal  year  are  thought  of  as  a  sample 
drawn  from  the  universe  of  airmen  whose  year  at  risk  covers  the  beginning  of  that  fiscal 
year,  that  sample  will  not  be  truly  representative  of  that  universe.  It  will  not  include 
airmen  who  have  already  left  or  who  have  already  reenlisted,  but  it  will  contain  all  the 
airmen  who  will  eventually  extend. 

The  bias  problem  is  most  severe  for  first-term  ETS  airmen.  To  illustrate  it,  in  a  10 
percent  sample  for  fiscal  year  1984,  of  the  airmen  whose  first-term  ETS  year  at  risk 
covered  the  beginning  of  FY  84,  more  than  half  of  the  reenlistments  had  already 
occurred  before  the  fiscal  year  began.  However,  most  of  the  losses  occurred  after  the 
fiscal  year  began.  The  loss  rate  for  this  year  at  risk  is  about  36  percent — the  rate 


estimated  by  the  cohort  loss  model.  The  loss  rate  for  the  fiscal  year  sample  is  about  42 
percent — 6  percentage  points  higher  than  the  cohort  loss  rate.  If  these  early  reenlistees 
were  characteristically  different  from  the  later  ones,  the  surviving  sample  might  be 
sufficiently  different  from  the  original  universe  that  it  would  drive  up  the  estimated  loss 
rates.  But  these  early  reenlistees  are  apparently  not  much  different  from  the  later  ones: 
unadjusted,  a  loss  rate  would  be  assigned  to  the  fiscal  year  sample  that  is  quite  close  to 
what  is  assigned  to  the  entire  universe,  thus  sharply  underestimating  the  actual  fiscal  year 
loss  rate. 

A  second  problem  involves  the  question  of  how  to  calculate  a  part-year  loss  rate 
using  the  whole-year  loss  rates  provided  by  the  cohort  loss  models.  There  are  two 
situations  where  this  problem  has  a  fairly  easy  solution.  In  the  first  situation,  if  one  can 
assume  all  the  losses  occur  at  the  end  of  the  year  at  risk,  then  the  part- year  loss  rate 
depends  on  whether  the  part  contains  the  end  of  the  year  at  risk:  if  it  contains  the  end, 
the  part-year  loss  rate  is  the  same  as  the  year-at-risk  rate;  if  it  doesn’t  contain  the  end,  the 
loss  rate  is  zero.  In  the  second  situation,  if  one  can  assume  losses  are  uniformly 
distributed  throughout  the  year,  there  is  a  fairly  simple  formula  (described  below)  for  the 
part- year  loss  rate.  Unfortunately,  few  if  any  models  correspond  to  either  of  these 
situations.  Although  ETS  losses  are  concentrated  at  year’s  end,  some  occur  earlier. 
Attrition  losses  can  happen  at  any  time,  but  not  always  uniformly  over  the  year.  The 
distribution  over  time  of  losses  from  extension  appears  to  be  quite  complicated,  perhaps  a 
mixture  of  losses  concentrated  at  the  end  with  losses  that  can  occur  anytime. 

Beginning  Survival  Rates.  This  section  explains  how  the  blending  model 
calculates  the  beginning  survival  rate  (denoted  SURVIVE_BEG  in  the  formula  given 
above).  Airmen  who  will  reach  retirement  eligibility  within  a  year  rarely  leave  before 
that  eligibility  is  reached.  For  such  airmen,  the  beginning  survival  rate  is  set  equal  to  1. 

Figure  3  shows  the  relationship  between  the  beginning  YAR  and  a  fiscal  year.  It 
breaks  the  beginning  YAR  into  two  parts:  an  early  part  (that  occurred  before  the  fiscal 

year  began,  denoted  "I - "),  and  what  has  already  been  called  the  beginning  risk 

interval  (the  part  that  covers  the  beginning  of  the  fiscal  year,  denoted  "  ===> ").  We  can 
write: 


SURVIVE_Y AR  =  SURVIVE_EAR  *  SURVIVE_BEG 
where 

SURVIVE_YAR  =  Survival  rate  for  the  beginning  YAR 
SURVIVE_EAR  =  Survival  rate  for  the  early  part  of  the  YAR 
SURVIVE_BEG  =  Survival  rate  for  the  beginning  risk  interval 


From  this,  it  follows  that 

SURVIVEJBEG  =  SURVIVE_YAR/SURVIVE_EAR 

The  beginning  cohort  loss  model  provides  an  estimate  of  the  year-at-risk  survival  rate, 
SURVIVE_YAR.6  All  that  is  needed  is  an  estimate  of  the  early  survival  rate 
(SURVIVE  EAR)  for  each  airman. 

Early  survival  rates  are  estimated  by  partitioning  the  airman  sample  into  "early 
groups,"  and  a  survival  rate  is  calculated  for  each.  Early  groups  are  defined  as  follows: 
First-term  attrition  airmen  are  divided  into  first-year  airmen,  last  attrition-year  airmen, 
and  all  the  rest.  Extenders  are  divided  into  decisionmakers  and  nondecisionmakers.  The 
other  early  groups  coincide  with  the  remaining  cohort  loss  models.  Thus  there  are  15 
grr  tps: 

First-term  attrition,  first  year 
First-term  attrition,  middle  years 
First-term  attrition,  last  year 
First-term  ETS 

First-term  extension,  nondecisionmakers 
First-term  extension,  decisionmakers 
Second-term  attrition 
Second-term  ETS 

Second-term  extension,  nondecisionmakers 
Second-term  extension,  decisionmakers 
Career-term  attrition 
Career-term  ETS 

Career-term  extension,  nondecisionmakers 
Career-term  extension,  decisionmakers 
Retirement 


"Actually,  it  provides  an  estimate  of  the  year-at-risk  loss  rate.  Subtracting  this  from  1 
gives  the  survival  rate. 


Each  of  these  groups  is  stratified  by  the  number  of  months  remaining  between  the 
beginning  of  the  fiscal  year  and  the  end  of  the  beginning  year  at  risk.  This  stratifies  each 
group  into  12  strata,  giving  180  early  groups  in  all.  Early  survival  rates  are  then 
calculated  from  the  same  YAR  file  during  the  merge  step  of  the  evaluation  run. 

Beginning  Extend-Given-Stay  Rates.  This  subsection  explains  how  the  test  and 
evaluation  blending  model  calculates  an  extend-given-stay  rate  that  applies  to  the  airmen 
found  at  the  beginning  of  the  fiscal  year.  The  rate  used  is  denoted  XGS  in  the  formula 
above.  Airmen  on  extension  status  and  ETS  airmen  are  considered. 

The  cohort  loss  models  do  not  estimate  extend-given-stay  rates  for  airmen  on 
extension  status.  However,  about  a  quarter  of  all  nondecisionmakers  reenlist  during  the 
first  extension  year,  making  the  extend-given-stay  rate  about  75  percent.  Because  there 
was  no  model  to  estimate  extend-given-stay  rates  for  such  airmen,  one  rate  was  estimated 
for  each  term  type  (first  term,  second  term,  and  career  term)  with  a  sample  of  airmen  on 
extension  status  in  FY  84.7 

For  ETS  airmen,  the  extend-given-stay  rate  provided  by  the  cohort  loss  model 
cannot  be  used  without  some  adjustment.  Perhaps  an  example  best  explains  why. 
Consider  a  10  percent  sample  of  first-term  ETS  airmen  in  FY  84,  containing  4,563 
airmen.  Losses,  extensions,  and  reenlistments  before  and  after  the  fiscal  year  were  as 
follows: 


Lost 

Extend 

Reenlist 

Stay 

Extend-Given- 
Stay  Rate 

Before  FY  began 

242 

0 

894 

894 

After  FY  began 

1415 

1200 

812 

2012 

60 

Entire  YAR 

1657 

1200 

1706 

2906 

4? 

Because  more  than  half  the  reenlistments  but  none  of  the  extensions  occurred  before  the 
fiscal  year  began,  the  group  of  airmen  who  entered  FY  84  had  a  much  higher  extend- 
given-stay  rate  than  the  cohort  as  a  whole.  These  models  do  not  appear  to  distinguish 
between  those  who  reenlist  early  and  those  who  do  not;  and  when  they  arc  applied  to  the 

Specifically,  for  a  10  percent  sample  of  airmen  in  FY  84  (SSN4),  extend-given-stay 
rates  were  76.0  percent  (first  term),  76.2  percent  (second  term)  and  73. 1  percent  (career 
term). 


fiscal-year  cohort,  they  give  estimates  similar  to  what  they  give  for  the  year-at-risk 
cohort.  Hence,  in  this  case,  they  sharply  underestimate  the  actual  extend-given-stay  rate. 

The  following  formal  analysis  shows  how  to  adjust  the  extend-given-stay  rate 
from  the  cohort  loss  model.  First  of  all, 

EXTENSIONS  =  EXTEND_GIVEN_STAY  * 

(NUM_AIRMEN  -  NUM_LOSSES) 

where 

EXTENSIONS  =  Number  of  extensions  for  the  YAR 

EXTEND_GIVEN_STAY  -  Extend  given-stay-rate  for  YAR 

NUM_AIRMEN  =  Number  of  airmen 

NUM_LOSSES  =  Number  of  losses 

All  extensions  occur  at  the  end  of  the  year  at  risk,  after  the  fiscal  year  begins.  The 
formula  that  estimates  these  extensions  in  terms  of  the  number  of  "stayers"  seen  among 
the  fiscal  year  cohort  is 

EXTENSIONS  =  XGS  *  (NUM_ A IRMEN  -  NUM_LOSSES  - 

NUM_EARLY_REENLIST) 

where 

XGS  =  Extend  given  stay  rate  for  FY 

NUM_EARLY_REENLIST  =  Number  of  reenlistments  prior  to  beginning  of  FY 

Equating  the  right-hand-sides  and  dividing  by  the  sample  size  to  obtain  rates  yields 

XGS  =  EXTEND_GIVEN_STAY*SURVIVE_YAR/ 

(SURVI  VE_Y  AR-REENLIST_EAR) 

where 

REENLIST_EAR  =  Early  reenlistment  rate 

The  cohort  loss  model  is  used  to  estimate  SURVIVE_YAR.  Values  of  REENLIST_EAR 
are  calculated  for  each  "early  group"  from  the  YAR  file  that  provides  the  evaluation 
sample  during  the  merge  step  of  die  evaluation  run  (at  the  same  time  as  the  early  survival 
rates  are  calculated,  as  described  above). 

Notice  that  the  adjustment  formula  mixes  two  estimates  from  the  loss  models  with 
an  actual  data  element,  REENLIST_EAR.  Nothing  in  the  formula  prevents  XGS  from 
exceeding  1 — this  could  happen  if  the  number  of  early  reenlistments  is  larger  than  the 


models  would  have  predicted  for  the  whole  year.  In  the  blending  code,  XGS  is  truncated 
so  that  it  never  exceeds  1. 

Ending  Survival  Rates.  This  section  explains  how  the  blending  model  calculates 
the  ending  survival  rates,  denoted  SURVIVE_REN  and  Sl)RVIVE_EXT  in  the  blending 
formula. 

If  the  airman  ends  up  in  an  ETS  year  at  risk,  the  ETS  decision  will  not  take  place 
until  the  next  fiscal  year,  so  only  attrition  losses  can  occur  during  the  current  fiscal  year. 
The  loss  rate  provided  by  the  beginning  attrition  model  is  used  to  estimate  this 
subsequent  attrition  loss  rate.  Thus,  SURVIVE_REN  is  always  based  on  an  attrition 
model. 

The  distribution  of  attrition  losses  over  the  year  at  risk  appears  to  be  complicated, 
but  to  calculate  ending  risk  interval  survival  rates  the  simplification  has  been  adopted  that 
they  are  uniformly  distributed  over  time.  This  hypothesis  leads  to  a  simple  formula  for 
the  survival  rate  over  any  part  of  a  year  as  follows. 

Suppose  the  survival  rate  for  a  year  at  risk  is  denoted  SURVIVEJYAR,  and  the 
daily  survival  rate  is  denoted  SURVIVE_ONE_DAY.  By  the  assumption  of  uniformity, 
the  survival  rate  is  the  same  every  day  of  the  year,  and 

365 

SURVIVE_YAR  =  SURVIVE_ONE_DAY 

Then  the  probability  of  surviving  N  days,  denoted  SURVIVE_DAY(N).  is 

N 

SURVI  VE_DA  Y  (N)  =  SURVIVE_ONE_DAY 
(N/365) 

=  SURVIVE_Y  AR 

The  probability  of  surviving  some  fraction  F  of  a  year, 

(0  <-=  F  <=  1),  denoted  SURVIVE_FRA(F),  is 
F 

SURVIVE_FRA(F)  =  SURVIVE_YAR 
This  formula  is  used  for  both  extending  and  reenlisting  survival  rates. 


If  the  extending  model  is  an  extension  model,  the  extending  model  loss  rate  is 
used,  just  as  an  attrition  model  does.  This  assumption  is  certainly  appropriate  for 
nondecisionmakers,  where  losses  would  be  similar  to  attrition  losses.  The  assumption 
may  seem  inappropriate  for  decisionmakers,  who  should  be  similar  to  airmen  in  their 
ETS  years.  However,  the  distribution  of  losses  over  the  year  at  risk  for  extension 
decisionmakers  is  quite  different  from  that  for  ETS  airmen;  the  losses  are  not 
concentrated  at  the  end  of  the  year,  but  are  much  more  spread  out.  For  one-year 
extensions,  the  loss  rates  seem  quite  uniformly  distributed  over  the  year,  except  for 
concentrations  at  3,  6,  9,  and  12  months.  For  two-year  extensions,  losses  concentrate 
substantially  at  the  very  end  of  the  two-year  term,  but  earlier  losses  are  much  higher  than 
observed  in  attrition  models,  so  the  "trick"  used  for  the  ETS  model  does  not  seem 
appropriate.  Assuming  uniform  distribution  seems  the  best  compromise,  given  what  is 
known  at  this  time.  This  problem  deserves  more  research. 

Step  5:  Compare.  The  process  described  above  yields  a  file  containing  a  record 
for  each  airman  found  at  the  beginning  of  the  fiscal  year.  The  file  contains  an  airmen 
identifier  and  any  other  characteristics  desired  (grade,  AFSC,  etc.).  With  such  a  file,  the 
number  of  losses  over  the  fiscal  year  may  be  calculated  for  any  group  of  airmen  by 
simply  adding  up  the  loss  probabilities  over  all  airmen  in  that  group. 

To  compare  the  estimates  with  what  actually  happened,  the  YAR  file  is  read  again, 
this  time  with  years  at  risk  that  cover  the  beginning  of  the  next  fiscal  year.  All  the 
airmen  are  sought  who  were  in  the  original  sample.  Airmen  whose  records  cannot  be 
found  are  assumed  lost.  Airmen  whose  records  indicate  that  they  left  before  the  new 
fiscal  year  began  but  who  were  not  early-outs  are  also  assumed  lost.  This  process  yields 
another  file  containing  one  record  for  every  airmen  in  the  original  sample  and  a  dummy 
variable  set  equal  to  0  if  the  airman  is  not  in  the  service  at  the  beginning  of  the  new  fiscal 
year,  and  1  if  he  is. 

When  these  two  files  are  combined,  it  is  possible  to  calculate  the  actual  size  of  any 
definable  group.  A  comparison  of  actual  with  predicted  sizes  permits  an  evaluation  of 
the  model  performance. 


MODEL  PERFORMANCE 

This  section  discusses  the  performance  of  the  blended  loss  models.  The  models 
were  calibrated  with  data  through  FY  83.  Data  are  available  only  through  FY  85,  and  the 
blended  models  require  two  fiscal  years  of  data.  Therefore,  only  FY  84  is  discussed.  It 
is  the  only  year  for  which  a  true  evaluation  of  the  blended  loss  models  can  be  made;  any 
earlier  year  would  require  using  data  that  were  also  used  for  calibration,  and  data  for 
later  years  were  not  available  at  the  time  of  the  analysis. 

The  discussion  is  in  two  parts.  First  the  overall  performance  of  the  models  is 
shown.  Then  the  performance  of  the  three  worst-performing  models  is  discussed  in  order 
to  discover  the  sources  of  the  errors  so  that  they  can  be  improved. 

Performance  Overview 

Table  3.1  summarizes  the  performance  of  the  blended  models,  stratifying  the 
sample  by  the  beginning  model  and  "early  group"  defined  earlier. 

Out  of  a  sample  of  nearly  48,000  airmen,  the  models  combine  to  overpredict 
losses  by  about  200  airmen,  with  a  relative  error  of  just  over  4  percent.  Although  this  is 

Table  3.1 

BLENDED  PERFORMANCE  BY  BEGINNING  MODEL  AND  GROUP 


Beginning 

Model/Group 

Airmen 

at 

Risk 

Loss  Rate  (%) 

Predicted  Actual 

PE 

PRE  (%) 

PEPNR  (%) 

SPE 

All 

47775 

11.4 

10.9 

217 

0.5 

4.2 

3.1 

1ATT  lstyr 

5798 

9.1 

6.0 

184 

3.2 

53.3 

8.4 

1ATT  Middle 

6999 

9.3 

8.4 

64 

0.9 

10.9 

2.6 

1ATT  Lastyr 

5187 

6.3 

8.1 

-94 

-1.8 

-22.3 

-5.4 

1ETS 

3488 

51.1 

46.9 

147 

4.2 

9.0 

5.0 

1EXT  Att 

479 

20.5 

18.6 

9 

1.9 

10.1 

1.0 

1EXT  DOS 

776 

41.1 

41.6 

-4 

-0.5 

-1.2 

-0.3 

2ATT 

8794 

2.2 

4.3 

-181 

-2.1 

-47.9 

-13.0 

2ETS 

1041 

23.3 

17.1 

65 

6.2 

36.5 

4.8 

2EXT  Att 

337 

9.2 

5.0 

14 

4.2 

82.4 

2.6 

2EXT  DOS 

201 

25.9 

16.9 

18 

9.0 

52.9 

2.9 

CATT 

9605 

1.7 

1.6 

5 

0.1 

3.2 

0.4 

CETS 

1531 

2.8 

2.9 

-2 

-0.1 

-4.4 

-0.3 

CEXT  Att 

283 

3.5 

2.5 

3 

1.1 

42.9 

1.0 

CEXT  DOS 

168 

8.9 

5.4 

6 

3.6 

66.7 

1.6 

RET 

3088 

31.3 

31.9 
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-1.7 

-0.7 
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low,  the  standardized  error  of  about  3.0  is  too  high  to  support  the  hypothesis  that  the 
overprediction  is  due  to  chance  alone. 

In  seven  of  the  16  model/groups,  the  absolute  value  of  the  standardized  prediction 
error  is  less  than  2.0,  suggesting  that  for  these  groups  the  discrepancy  between  actual  and 
predicted  losses  is  due  to  chance.  In  particular,  the  career  model/groups  and  the 
retirement  model  predictions  are  all  quite  accurate;  these  groups  combined  account  for 
nearly  a  third  of  the  sample.  However,  career  model  loss  rates  are  very  low.  Of  the 
models  with  moderate  to  high  loss  rates,  the  retirement  model  does  the  best  job,  with  a 
standardized  prediction  error  well  within  the  range  that  could  be  expected  to  occur  by 
chance  alone. 

Large  errors  can  be  seen  in  the  first-  and  second-term  attrition  model  groups,  and 
in  first-term  ETS.  The  next  section  will  show  that  these  errors  seem  to  be  due  to 
problems  in  the  component  cohort  loss  models,  although  the  possibility  that  they  are  also 
due  to  errors  in  the  blending  model  cannot  be  ruled  out. 

Detailed  Analysis 

The  blended  loss  rates  result  from  a  combination  of  the  cohort  loss  models  and  the 
blending  model.  The  fiscal  year  evaluation  is  as  much  a  test  of  the  first  effort  as  it  is  of 
the  second.  But  there  is  an  asymmetry  in  the  two  groups  of  models:  The  cohort  loss 
models  can  be  tested  without  the  blending  models,  but  the  blending  models  cannot  be 
tested  without  the  cohort  loss  models.  This  asymmetry  constrains  the  ability  to  test  the 
blending  model:  Only  when  the  cohort  loss  model  is  eliminated  as  a  source  of  error  can 
the  blending  model  be  tested. 

The  sharp  differences  between  predicted  and  actual  loss  rates  for  the  first-  and 
second-term  attrition  models  and  for  first-term  ETS  were  rated  above.  These  differences 
challenge  the  accuracy  of  either  the  blending  model  or  of  the  component  cohort  loss 
models  (or  both).  But  before  it  is  possible  to  analyze  the  blending  model  as  a  source  of 
the  problem,  errors  should  be  eliminated  from  the  cohort  loss  models.  In  what  follows,  it 
will  be  shown  that  the  discrepancies  are  largely  due  to  the  cohort  loss  models;  therefore, 
such  errors  in  the  blending  model  as  remain  cannot  be  revealed  by  merely  examining 
performance. 

For  first-term  attrition,  it  is  necessary  merely  to  discuss  airmen  in  their  first  and 
last  year,  as  the  story  for  the  middle  year  is  much  the  same. 
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First- Year,  First-Term  Attrition 

Table  3.2  presents  an  accounting  of  predicted  and  actual  losses  for  a  sample  of 
airmen  whose  first  year  of  their  first  term  covered  the  beginning  of  FY  84.  The 
beginning  of  this  year  coincided  with  the  beginning  of  the  fiscal  year  for  only  a  small 
fraction  of  airmen,  so  of  the  6,220  airmen  in  the  sample,  some  422  had  left  the  service 
before  the  fiscal  year  began,  leaving  5,798  at  the  beginning  of  the  fiscal  year.  It  is  these 
5,798  airmen  to  whom  the  blended  loss  model  applies. 

The  loss  rates  displayed  in  Table  3.2  are  not  annual  loss  rates;  they  are  simply  the 
ratios  of  the  losses  to  the  number  at  risk  expressed  as  a  percent.  The  time  at  risk  for  each 
group  is,  on  the  aveiage,  six  months.  Notice  that  the  actual  loss  rate  before  the  beginning 
of  the  fiscal  year  (6.8  percent)  is  much  higher  than  the  actual  loss  rate  afterward  (3.0 
percent)  because  the  tendency  to  drop  out  in  the  first  half  of  the  first  year  of  service  is 
greater  than  in  the  second  half. 


The  loss  rate  for  the  entire  first  year  at  risk  is  (422  +  175)  divided  by  6,220,  or  9.6 
percent.  That  is  the  number  that  the  cohort  loss  model  attempts  to  predict.  Table  2.5  for 
FY  84  showed  that  the  predicted  cohort  loss  rate  for  airmen  in  their  first  year  of  service  is 
1 1.0  percent.8  Therefore,  the  predicted  cohort  loss  rates  are  too  high.  Such  a  rate  applied 
to  the  6,220  airmen  in  the  year  at  risk  would  yield  a  predicted  684  losses.  Subtracting  the 
422  actual  early  losses  leaves  262  predicted  losses  for  the  first  part  of  the  fiscal  year,  for 
a  predicted  loss  rate  in  the  first  part  of  the  fiscal  year  of  4.5  percent,  quite  close  to  the  rate 
of  4.8  percent  seen  in  Table  3.2.  The  discrepancy  between  actual  and  predicted  loss  rates 


Table  3.2 


PREDICTED  AND  ACTUAL  LOSSES,  FIRST-YEAR,  FIRST-TERM  ATTRITION 

(Loss  rates  in  parentheses) 


Actual 

Predicted 

Beginning  of  YAR: 

Early  losses 

6,220 

422  (6.8) 

Beginning  of  FY: 

Remaining  YAR  losses 

5,798 

175  (3.0) 

5,798 

281 

(4.8) 

Beginning  of  next  YAR: 

Early  losses 

5,623 

170  (3.0) 

5,517 

248 

(4.5) 

8Table  2.5  showed  an  actual  loss  rate  of  9.4  percent,  which  is  not  substantially 
different  from  the  rate  of  9.6  percent  seen  in  Table  3.2. 
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in  the  first  half  of  Fr  84  appears  to  be  due  largely  to  ail  overpreuiclion  in  the  cohort  loss 
model. 

Losses  in  the  second  half  of  FY  84  should  be  compared  with  the  performance  of 
the  cohort  loss  model  for  airmen  in  their  second  year  of  service  for  FY  85.  Table  2.5 
showed  that  the  actual  loss  rate  for  such  airmen,  7.1  percent,  is  somewhat  lower  than  the 
predicted  loss  rate  of  9.0  percent.  Compared  with  Table  3.2,  these  rates  have  to  be 
adjusted  to  the  time  at  risk  of  six  months.  Assuming  that  losses  are  uniform  over  the 
second  year,  adjusting  the  loss  rates  in  Table  2.5  to  six  months  gives  a  predicted  loss  rate 
of  4.6  percent  and  an  actual  loss  rate  of  3.6  percent.  This  explains  most  of  the  difference 
between  loss  rates  seen  in  the  bottom  line  of  Table  3.2,  although  it  suggests  that  the 
assumption  built  into  the  blending  code  that  losses  are  uniform  over  time  may  not  be 
quite  correct. 

Blended  predicted  losses  for  first-year  airmen  are  sharply  higher  than  the  loss  rates 
actually  observed.  This  overprediction  appears  to  be  caused  by  the  cohort  loss  models’ 
overpredicting  losses  for  first-  and  second-year  airmen. 

Last-Year,  First-Term  Attrition 

Table  3.3  presents  an  accounting  of  predicted  and  actual  losses  for  a  sample  of 
airmen  whose  last  first-term  attrition  year  covered  the  beginning  of  FY  84.  Table  3.3 
will  not  be  analyzed  as  closely  as  Table  3.2.  Table  3.1  noted  that  the  predicted  loss  rate 
for  FY  84  was  somewhat  lower  than  the  actual  loss  rate  for  these  airmen.  Here  the  major 
source  of  this  difference  appears  to  be  due  to  the  losses  seen  in  the  first  part  of  FY  84, 

Table  3.3 

PREDICTED  AND  ACTUAL  LOSSES,  LAST- YEAR,  FIRST-TERM  ATTRITION 

(Loss  rates  in  parentheses) 


Actual  Predicted 


Beginning  of  Y AR:  5,420 

Early  losses  230  (4.2) 

Early  reenlistments  3 

Beginning  of  FY:  5,187  5,187 

Remaining  YAR  losses  267  (5.1)  177  (3.4) 

Reenlistments  74  0 

Beginning  of  next  YAR:  4,463  5,010 

Early  losses  159  (3.6)  150  (3.0) 
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where  the  actual  loss  rate  is  5.1  percent;  the  predicted  loss  rate  is  just  3.4  percent.  Most 
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of  these  airmen  are  in  their  third  year  of  service,  and  from  Table  2.5  for  such  airmen,  the 
annual  loss  rate  predicted  by  the  cohort  loss  model,  7. 1  percent,  falls  somewhat  below 
the  actual  loss  rate  of  9.4  percent.  Thus,  the  difference  seen  here  is  accounted  for  by  the 
cohort  loss  model. 

First-Term  ETS 

Table  3.1  showed  a  predicted  fiscal  year  loss  rate  of  51.1  percent,  sharply  higher 
than  the  actual  rate  of  46.9  percent,  for  first-term  ETS  airmen.  The  model  predicts  146 
too  many  losses.  Table  3.4  presents  an  accounting  of  the  sample  used  to  calculate  these 
tables.  It  is  a  complicated  table,  but  the  predicted  number  of  losses  in  the  beginning  of 
the  fiscal  year  exceeds  the  actual  number  by  146  airmen. 

Table  2.3  showed  that  the  predicted  cohort  loss  rates  for  first-term  ETS  airmen  in 
FY  84  is  38.9  percent,  while  the  actual  rate  is  36.5  percent.  These  rates  apply  to  a  year  at 
risk,  not  to  a  fiscal  year,  cohort.  Applying  this  rate  to  the  4,563  airmen  in  the  entire  year 
at  risk  yields  an  estimated  1,775  losses,  and  if  the  242  early  losses  are  subtracted,  1,533 


Table  3.4 

PREDICTED  AND  ACTUAL  LOSSES,  FIRST-TERM  ETS 
(Loss  and  extend-given-stay  rates  are  in  parentheses) 


Actual 

Predicted 

Predicted- 

Actual 

Beginning  of  YAR: 

4,563 

Early  outs 

61 

Other  early  losses 

181 

Early  losses 

242 

(5.3) 

Early  recnlistmcnts 

894 

Beginning  of  FY: 

3,427 

3,427 

146 

Remaining  YAR  losses 

1,415 

(41.3) 

1,561 

(45.6) 

"Stayers" 

2,012 

1,866 

Extensions 

1,200 

(59.6) 

1,275 

(68.3) 

75 

Short  extensions 

606 

595 

Losses 

150 

(24.8) 

140 

(23.5) 

-10 

Long  extensions 

594 

680 

Losses 

12 

(2.0) 

17 

(2.5) 

5 

Reenlistments 

812 

591 

Losses 

9 

(LI) 

2 

(0.3) 

-7 

t.  f.  A  i 


losses  are  predicted  to  occur  after  the  fiscal  year  began.  The  number  of  losses  predicted 
by  the  blended  model  exceeded  this  number  by  28  airmen.  This  part  of  the  error  may  be 
charged  to  the  blending  model,  although  sampling  error  may  be  some  small  pan  of  it. 
The  year  at  risk  sample  has  been  reduced  through  losses  and  reenlistments  by  25  percent 
by  the  time  the  fiscal  year  begins.  The  blending  model  attempts  to  adjust  the  loss 
predictions  for  these  early  losses  but  may  not  go  far  enough. 

To  summarize,  the  blended  model  overestimates  losses  before  the  end  of  the  year 
at  risk  by  146.  Of  the  146  overestimated,  118  may  be  due  to  the  cohort  loss  model  and 
28  may  be  due  to  the  blending  model.  In  percentage  terms,  about  81  percent  of  the 
overprediction  may  be  due  to  the  cohort  loss  model. 

There  is  a  sharp  difference  between  the  actual  and  predicted  extend-given-stay 
rates  for  the  airmen  who  survived  to  the  beginning  of  the  fiscal  year.  This  difference 
appears  to  be  due,  at  least  in  part,  to  a  similar  difference  in  the  predicted  and  actual 
cohort  extend-given-stay  rates,  as  reported  in  Table  2.4.  Had  the  two  extend-given-stay 
rates  been  closer,  the  net  discrepancy  would  have  been  reduced  by  about  20  airmen. 
Although  improving  the  cohort  extend-given-stay  model  is  a  goal  worth  pursuing, 
improving  the  loss  model  is  much  more  important.9 

Subsequent  to  the  test  and  evaluation  of  this  model,  a  new  and  greatly  improved 
first-term  ETS  loss  model  was  estimated. 


IV.  COMPARISON  WITH  ALPS 


INTRODUCTION 
The  ALPS  Model 

The  Airman  Loss  Probability  System  (ALPS)  provides  projected  loss  rate 
estimates  to  the  Airmen  Skill  Force  Model  (ASKIF),  which  is  the  principal  tool  currently 
being  used  by  the  Air  Force  to  estimate  the  annual  requirements  for  trained  enlisted 
personnel  by  Air  Force  specialty  (AFSC).  ALPS  will  be  maintained  until  it  is  replaced 
by  the  EFMS  middle-term  disaggregate  loss  models  discussed  here. 

ALPS  calculates  loss  rates  using  data  from  the  preceding  12  months.  It  partitions 
the  previous  year’s  inventory  into  several  classes,  calculates  the  loss  rate  within  each 
class,  and  then  assumes  that  the  same  loss  rates  will  be  observed  within  each  class  for  the 
current  inventory. 

In  any  of  these  classes,  the  ALPS  prediction  error  can  be  expressed  in  the  form: 

ERROR  =  (P(y)  -  P(y  -  1))  *  N(y) 
where 

N(y)  =  number  of  airmen  in  the  class 
P(y)  =  this  year’s  loss  rate  for  the  class 

P(y-l)  =  last  year’s  loss  rate  for  the  class 

The  ALPS  model  can  be  sensitive  to  distributional  changes  in  the  force  (changes  in  the 
number  of  airmen  in  AFSCs,  grade  levels,  etc).  If  from  one  year  to  the  next  a  large 
movement  of  airmen  were  to  occur  from  a  low-loss  rate  class  to  a  high-loss  rate  class, 
ALPS  could  accurately  track  the  effects  of  that  change.  ALPS  is  insensitive  to  changes 
in  the  underlying  loss  rate  that  might  be  caused  by  changes  in  external  circumstances, 
however,  such  as  the  unemployment  rate,  the  military/civilian  pay  ratio,  etc.  If  from  one 
year  to  the  next  the  unemployment  rate  were  to  fall  sharply  and  civilian  pay  were  to  rise 
sharply  compared  with  military  pay  in  some  occupational  class,  ALPS  would  not  be  able 
to  predict  the  increase  in  losses  that  might  occur  because  of  these  shifts  in  the  economy. 


Comparison  Years 

EFMS  was  compared  with  ALPS  for  FY  82  and  FY  84.  FY  84  is  the  only  fiscal 
year  for  which  data  are  currently  available  that  were  not  used  for  the  estimation. 
However,  FY  84  is  a  somewhat  unfortunate  year  for  making  a  comparison.  ALPS  uses 
the  previous  fiscal  year  to  calculate  loss  rates.  When  there  is  little  change  in  the 
economy  or  in  Air  Force  compensation  policies  from  one  year  to  the  next,  both  ALPS 
and  EFMS  should  do  reasonably  well  and  produce  similar  loss  predictions.  In  those 
aspects  of  change  to  which  the  EFMS  model  is  sensitive  (unemployment  and  wages),  FY 
83  and  FY  84  were  quite  similar,  as  the  following  shows: 

FY  83  FY  84 

Unemployment  rate  (ages  20-24)  14.8  14.2 

Military/Civilian  wage  ratio  0.943  0.940 

On  the  basis  of  this  similarity,  the  performance  of  ALPS  and  the  EFMS  models  should  be 
quite  similar,  and  ii  was. 

A  quick  scan  of  other  years  suggested  that  FY  82  might  show  a  different  picture. 
Economic  changes  were  indeed  greater,  as  the  following  shows: 


FY  81 

FY  82 

Unemployment  rate 

11.5 

12.2 

Military/Civilian  wage  ratio 

0.953 

0.993 

EFMS  was  compared  with  ALPS  for  FY  82  also,  although  data  from  FY  82  were  used  to 
estimate  the  EFMS  models.  However,  FY  82  was  not  the  only  year  used  to  estimate  the 
EFMS  models,  but  was  only  one  of  seven  such  years. 


Data  Processing 

Tapes  containing  FY  82  and  FY  84  ALPS  estimates  were  obtained  from  the  Air 
Force.  Airman  records  on  the  ALPS  tapes  are  identified  by  the  airman’s  Social  Security 
Number  (SSN),  but  for  privacy  reasons  this  number  is  not  kept  on  the  YAR  files  used  for 
estimation  and  evaluation.  The  ALPS  files  were  merged  with  a  cross-walk  file,  and  files 
were  created  containing  the  ALPS  estimates  in  which  the  airmen  were  identified  by  the 
YAR  variable  that  identifies  each  airman  (a  variable  called  BID).  Two  10  percent 
samples  were  set  aside,  one  for  FY  82  (corresponding  to  the  YAR  sample  in  which  the 
airman’s  SSN  ends  in  a  3)  and  one  for  FY  84  (corresponding  to  the  YAR  sample  in 
which  the  airman’s  SSN  ends  in  a  4). 

The  ALPS  records  contain  several  loss  rates.  The  one  called  Total  Loss 
Probability ,  which  is  the  probability  of  any  kind  of  loss,  attrition,  ETS,  etc.,  over  the 
fiscal  year  was  the  one  used. 

The  end  of  a  series  of  somewhat  complicated  steps  in  Sec.  Ill  yielded  a  file 
containing  one  record  for  each  airman  found  at  the  beginning  of  a  fiscal  year,  along  with 
his  EFMS  loss  probability,  an  indicator  as  to  whether  he  was  actually  in  the  force  at  the 
beginning  of  the  next  fiscal  year,  and  any  other  desired  characteristics,  including  his 
YAR  identifier,  BID.  It  is  a  technically  simple  matter,  therefore,  once  the  steps  described 
in  Sec.  Ill  are  complete,  to  merge  this  file  with  the  corresponding  ALPS  samples  just 
described,  and  obtain  a  file  with  the  ALPS  estimate  of  the  airman’s  loss  probability 
alongside  that  of  the  EFMS  estimate. 

Such  files  were  created  for  FY  84  and  FY  82  and  then  merged  with  the 
corresponding  ALPS  samples.  The  match  was  extremely  good.  Of  approximately 
50,000  cases  in  both  samples,  only  a  handful  of  records  could  not  be  matched,  either 
because  they  were  found  in  the  YAR  file  but  were  not  in  the  ALPS  file,  or  vice  versa. 

RESULTS  OF  THE  COMPARISON 

Tables  4.1  and  4.2  show  the  results  of  the  comparisons  for  FY  84  and  FY  82 
respectively,  presented  by  loss  model  category,  except  that  the  first  term  attrition  model 
is  partitioned  into  three  subgroups:  (1)  first- year  airmen;  (2)  "last- year  airmen" — by 
which  is  meant  the  last  attrition  year,  which  could  be  the  third  year  of  a  four-year  term  or 
the  fifth  year  of  a  six-year  term;  and  (3)  "middle  airmen" — neither  first  year  nor  last.  The 
categories  in  these  tables  are  the  same  as  those  of  Table  3.1. 
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Table  4.1 


EFMS-ALPS  COMPARISON,  FY  84 


Beginning  Airmen  Loss  Rates  (%)  PE  (%)  PEPNR  (%) 

Submodel  at 


Group 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

All 

47735 

11.3 

9.4 

121 

-1.5 

1  ATT 

1st  yr 

5792 

5.9 

9.1 

7.2 

184 

74 

3.2 

1.3 

1ATT 

Middle 

6999 

8.4 

9.3 

6.3 

64 

-149 

0.9 

-2.1 

1ATT 

Last  yr 

5183 

8.1 

6.3 

6.6 

-94 

-80 

-1.8 

-1.5 

1ETS 

3487 

46.9 

51.1 

42.0 

147 

-170 

4.2 

-4.9 

1EXT 

Att 

479 

18.6 

20.5 

19.4 

9 

4 

1.9 

1EXT 

DOS 

768 

42.1 

44.8 

-9 

21 

-1.2 

2.7 

2  ATT 

8785 

4.3 

2.2 

-177 

-108 

-2.0 

-1.2 

2ETS 

1040 

17.1 

23.4 

13.4 

65 

-39 

6.3 

-3.8 

2EXT 

Att 

337 

5.0 

9.2 

4.2 

14 

-3 

4.2 

2EXT 

DOS 

16.5 

19 

1 

9.5 

HI 

CATT 

9599 

1.6 

1.7 

1.1 

7 

-42 

0.1 

S9 

CETS 

1529 

2.9 

2.8 

-1 

2 

0.1 

CEXT 

Att 

283 

2.5 

3.5 

1.4 

3 

-3 

1.1 

-1.1 

CEXT 

DOS 

167 

4.8 

9.0 

4.2 

7 

-1 

4.2 

-0.6 

RET 

3087 

31.8 

31.3 

-17 

-0.6 

-6.8 

The  number  of  airmen  at  risk  in  Table  4. 1  is  slightly  lower  than  the  corresponding 
number  in  Table  3. 1  because  of  a  small  number  (40  in  all)  of  airmen  whose  ALPS 
records  could  not  be  matched  to  the  YAR  file.  This  number  is  very  small  compared  with 
the  total  (less  than  0.1  percent)  and  the  unmatched  cases  are  spread  seemingly  at  random 
over  all  the  models;  it  was  therefore  not  worth  the  effort  to  resolve  the  difference,  or 
even  find  out  why  they  differed.  In  addition,  a  handful  of  ALPS  records  could  not  be 
matched  to  the  YAR  file,  and  those  have  been  ignored  also. 

Table  4.1  shows  that  the  EFMS  model  comes  closer  overall  to  estimating  the  total 
number  of  FY  84  losses  than  does  ALPS.  EFMS  overestimates  the  total  number  of  losses 
by  221  airmen,  while  ALPS  underestimates  it  by  703  airmen.  Both  errors  are  larger  in 
absolute  value  than  one  would  expect  to  see  by  chance  alone:  The  standard  deviation  of 
an  unbiased  estimator  of  the  number  of  losses  is  about  68  airmen. 

Looking  at  specific  submodels  or  model  groups  in  Table  4. 1 ,  there  are  several 
exceptions  to  the  average  result  that  EFMS  overpredicts  losses  while  ALPS 
underpredicts  them.  Perhaps  the  most  notable  occurs  in  the  second-term  attrition  model, 
where  both  models  underpredict  the  losses,  with  EFMS  predicting  even  fewer  than 


i 

! 


EFMS-ALPS  COMPARISON,  FY  82 

Beginning  Airmen  Loss  Rates  (%)  PE  (%)  PEPNR  (%) 

Submodel  at  -  -  - 


Group 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

All 

43418 

9.4 

12.7 

13.7 

1454 

1884 

3.3 

4.3 

1  ATT 

1st  yr 

6595 

6.2 

9.6 

7.5 

222 

84 

3.4 

1.3 

1  AIT 

Middle 

5984 

7.2 

9.1 

7.1 

115 

-7 

1.9 

-0.1 

1  ATT 

Last  yr 

4441 

5.1 

8.6 

18.2 

155 

581 

3.5 

13.1 

1ETS 

3833 

36.2 

43.0 

50.9 

263 

564 

6.9 

14.7 

1EXT 

Att 

696 

13.8 

22.8 

29.2 

63 

107 

9.1 

15.4 

1EXT 

DOS 

775 

31.7 

46.2 

54.7 

112 

178 

14.5 

23.0 

2ATT 

5548 

1.9 

2.5 

3.3 

34 

80 

0.6 

1.4 

2ETS 

1044 

17.1 

23.4 

21.2 

65 

42 

6.2 

4.0 

2EXT 

Att 

488 

5.5 

9.6 

9.6 

20 

20 

4.1 

4.1 

2EXT 

DOS 

324 

13.6 

26.9 

20.1 

43 

21 

13.3 

6.5 

CATT 

8442 

0.8 

2.2 

1.7 

122 

80 

1.4 

0.9 

CETS 

1665 

3.4 

3.8 

5.9 

8 

42 

0.5 

2.5 

CEXT 

Att 

430 

1.2 

3.7 

3.7 

11 

11 

2.6 

2.6 

CEXT 

DOS 

213 

5.2 

9.4 

6.6 

9 

3 

4.2 

1.4 

RET 

2940 

26.7 

33.9 

29.3 

212 

78 

7.2 

2.7 

ALPS.  A  similar  event  can  be  seen  for  the  last  year  of  the  first-term  attrition  model.  But 
in  1 1  out  of  15  cases  in  Table  4.1,  ALPS  predicts  fewer  losses  than  EFMS.  In  seven  out 
of  the  15  cases,  the  ALPS  prediction  is  closer  to  what  actually  happened  than  the  EFMS 
prediction.  Generally  speaking,  the  two  models  perform  about  equally  well,  though 
neither  performs  extremely  well. 

Table  4.2  presents  the  same  comparison  for  FY  82.  Recall  that  this  was  a  year 
chosen  because  conditions  were  changing  more  rapidly  from  one  year  to  the  next:  The 
unemployment  rate  was  rising,  the  military/civilian  pay  ratio  was  rising,  and  loss  rates 
had  decreased  sharply  between  FY  81  and  FY  82.  Both  EFMS  and  ALPS  sharply 
overpredict  losses,  but  overall  EFMS  predicts  fewer  losses  than  ALPS.  For  the  first-term 
ETS  model,  the  ALPS  overprediction  is  twice  that  of  EFMS,  possibly  because  the  first- 
term  ETS  model  has  explicit  economic-sensitive  terms  that  reduce  loss-rate  predictions 
in  response  to  rising  unemployment  and  military  civilian  wage  ratios.  Flowever,  EFMS 
predicts  more  losses  than  ALPS  for  the  second-term  ETS  model,  which  also  contains 
such  terms. 


Table  4.2  suggests  that  while  EFMS  may  be  a  little  more  sensitive  to  annual 
changes  than  ALPS,  neither  is  sensitive  enough. 

Following  is  a  model-by-model  comparison  of  EFMS  with  ALPS,  and  of  both 
with  the  actual  results,  for  FY  82  and  FY  84.  There  are  10  sets  of  results,  one  for  each  of 
the  10  major  classes  of  middle-term  disaggregate  loss  models.  Within  ea<~h  set  are 
presented  tables  containing  detailed  comparisons  by  several  stratification  categories: 
bonus  level,  Career  Field  Group,'  grade,  term  of  enlistment,  term  year,  and  year  of 
service. 

The  tables  arc  meant  to  serve  as  benchmarks  for  future  evaluation  efforts.  So  the 
discussions,  if  any,  will  be  brief  and  meant  to  diagnose  possible  problems  that  remain 
either  with  the  EFMS  models  or  with  the  evaluation  effort  itself. 

First-Term  Attrition 

Tables  4.3  and  4.4  compare  EFMS  and  ALPS  for  the  first-term  attrition  model  for 
FY  82  and  FY  84  respectively. 

Table  4.3  reveals  something  strange  with  respect  to  the  bonus  level:  Losses  were 
practically  nonexistent  among  airmen  receiving  a  one-half  bonus.  ALPS  reflects  this 
phenomenon  slightly  with  a  little  dip  in  the  estimated  loss  rate;  but  EFMS  fails  and 
instead  predicts  higher  losses.  Nothing  like  that  happened  in  FY  84,  so  the  event 
probably  reflects  the  implementation  of  some  policy  that  was  in  place  in  FY  82  but  not  in 
FY  84. 

The  previous  section  pointed  out  that  EFMS  tended  to  overprcdict  FY  84  losses 
while  ALPS  tended  to  underprcdict.  This  pattern  can  be  observed  in  Table  4.4.  Little  in 
that  table  explains  why. 

First-Term  ETS 

Tables  4.5  and  4.6  compare  EFMS  and  ALPS  for  the  first-term  ETS  model  for  FY 
82  and  FY  84  respectively.  Both  EFMS  and  ALPS  exhibit  sizable  prediction  errors, 
especially  for  FY  82.  The  EFMS  first-term  ETS  model  has  been  rcspccificd  to  reduce  flic 
size  of  its  prediction  error. 

!Scc  Table  A.l  of  Carter  ct  al.  (1987)  for  definitions. 


Table  4.3 


FIRST-TERM  ATTRITION,  FY  82 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at 


Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Bonus 

0.0 

8703 

4.5 

9.5 

10.4 

439 

511 

5.0 

5.9 

0.5 

2032 

.4 

10.0 

7.0 

195 

134 

9.6 

6.6 

1.0 

2892 

6.7 

8.0 

12.6 

36 

170 

1.2 

5.9 

2.0 

918 

2.1 

6.8 

10.0 

43 

73 

4.7 

8.0 

3.0 

592 

6.3 

7.1 

11.0 

5 

28 

.8 

4.7 

Occupational 

CrfSer 

1709 

7.4 

13.2 

13.8 

98 

109 

5.7 

6.4 

Group 

ElMeRe 

2792 

6.4 

8.3 

12.3 

53 

164 

1.9 

5.9 

SkilTe 

3186 

5.8 

7.2 

11.6 

45 

185 

1.4 

5.8 

Suppor 

1953 

5.1 

8.3 

11.6 

64 

128 

3.3 

6.6 

Grade 

1 

2852 

9.0 

9.9 

7.7 

23 

-37 

.8 

-1.3 

2 

4806 

7.7 

8.6 

7.4 

45 

-13 

.9 

-.3 

3 

5931 

5.7 

9.1 

14.5 

204 

521 

3.4 

8.8 

4 

675 

3.1 

8.0 

12.3 

33 

62 

4.9 

9.2 

Term  of 

4 

15444 

6.3 

9.1 

10.3 

421 

612 

2.7 

4.0 

Enlistment 

6 

1541 

5.5 

10.1 

8.5 

72 

47 

4.7 

3.0 

Term  Year 

1 

6595 

6.2 

9.6 

7.5 

222 

84 

3.4 

1.3 

2 

5697 

7.3 

9.1 

7.1 

101 

-13 

1.8 

-.2 

3 

4507 

5.2 

8.8 

17.6 

162 

559 

3.6 

12.4 

5 

133 

4.5 

5.3 

21.1 

1 

22 

.8 

16.5 

Year  of 

0 

6626 

6.3 

9.6 

7.5 

215 

77 

3.2 

1.2 

Service 

1 

5677 

7.3 

9.1 

7.1 

103 

-11 

1.8 

-.2 

2 

4495 

5.1 

8.8 

17.6 

166 

564 

3.7 

12.5 

Stratification 

Factor 


Table  4.4 

FIRST-TERM  ATTRITION,  FY  84 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

at 

Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


Table  4.5 


FIRST-TERM  ETS,  FY  82 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at 


Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Bonus 

1.0 

1420 

36.3 

43.2 

52.5 

98 

231 

6.9 

16.3 

2.0 

218 

44.0 

42.7 

55.0 

_  2 

24 

-1.4 

11.0 

3.0 

167 

37.7 

45.5 

52.1 

13 

24 

7.8 

14.4 

Occupational 

CrfScr 

671 

38.2 

47.2 

50.4 

61 

82 

9.1 

12.2 

Group 

ElMcRe 

1193 

37.6 

42.9 

53.1 

63 

184 

5.3 

15.4 

SkilTc 

1262 

38.3 

45.5 

52.6 

91 

181 

7.2 

14.3 

Suppor 

687 

27.5 

34.2 

44.5 

46 

117 

6.7 

17.0 

Grade 

3 

137 

47.4 

46.7 

56.2 

-1 

12 

-.7 

8.8 

4 

3517 

35.6 

43.1 

50.9 

263 

536 

7.5 

15.2 

5 

126 

26.2 

35.7 

42.9 

12 

21 

9.5 

16.7 

Table  4.6 

FIRST-TERM  ETS,  FY  84 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at  - 

Factor  Level  Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


First-Term  Extension 


Tables  4.7  and  4.8  compare  EFMS  and  ALPS  for  the  first-term  extension  model 
for  FY  82  and  FY  84  respectively. 


Table  4.7 

FIRST-TERM  EXTENSION,  FY  82 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at 

Factor  Level  Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


FIRST-TERM  EXTENSION,  FY  84 


Stratification 


Airmen 

at 


Loss  Rate  (%) 


PE  (%) 


PEPNR  (%) 


Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Bonus 

0.0 

859 

30.0 

31.9 

34.0 

16 

34 

1.9 

4.0 

1.0 

264 

37.1 

35.2 

39.0 

-5 

5 

-1.9 

1.9 

Occupational 

CrfSer 

251 

29.1 

31.1 

34.7 

5 

14 

2.0 

5.6 

Group 

ElMeRe 

260 

37.3 

31.5 

36.9 

-15 

-1 

-5.8 

-.4 

SkilTe 

357 

40.3 

40.9 

38.4 

2 

-7 

.6 

-2.0 

Suppor 

364 

25.8 

26.9 

30.8 

4 

18 

1.1 

4.9 

Grade 

4 

950 

33.2 

32.7 

34.3 

—4 

11 

-.4 

1.2 

5 

255 

30.2 

34.1 

37.3 

10 

18 

3.9 

7.1 

Term  Year 

5 

812 

29.1 

31.5 

30.0 

20 

8 

2.5 

1.0 

6 

369 

41.5 

35.8 

47.2 

-21 

21 

-5.7 

5.7 

Year  of 

4 

814 

28.9 

31.3 

30.1 

20 

10 

2.5 

1.2 

Service 

5 

365 

41.6 

36.2 

47.4 

-20 

21 

-5.5 

5.8 

Second-Term  Attrition 

Tables  4.9  and  4.10  compare  EFMS  and  ALPS  for  the  second-term  attrition  model 
for  FY  82  and  FY  84  respectively. 

Second-Term  ETS 

Tables  4.1 1  and  4.12  compare  EFMS  and  ALPS  for  the  second-term  ETS  model 
for  FY  82  and  FY  84  respectively. 

Second-Term  Extension 

Tables  4.13  and  4.14  compare  EFMS  and  ALPS  for  the  second-term  extension 
model  for  FY  82  and  FY  84  respectively. 

Career  Attrition 

Tables  4.15  and  4.16  compare  the  career  attrition  loss  rates  from  the  EFMS  and 
ALPS  models  for  FY  82  and  FY  84  respectively.  In  both  years,  an  anomaly  can  be  seen 
for  airmen  in  their  17th  and  18th  years  of  service.  Although  the  actual  loss  rates  are 
quite  low  for  such  airmen,  both  EFMS  and  ALPS  predict  higher  losses.  The  higher 
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Tabic  4.9 

SECOND-TERM  ATTRITION,  FY  82 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at 

Factor  Level  Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


am 


Table  4.10 


SECOND-TERM  ATTRITION,  FY  84 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at  -  -  - 


Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Bonus 

0.0 

5534 

4.4 

2.3 

3.1 

-111 

-68 

-2.0 

-1.2 

0.5 

285 

6.0 

2.1 

3.2 

-11 

-8 

-3.9 

-2.8 

1.0 

2404 

3.8 

2.1 

2.8 

-42 

-24 

-1.7 

-1.0 

2.0 

247 

3.6 

2.0 

2.8 

-4 

-2 

-1.6 

-.8 

3.0 

227 

4.0 

1.8 

2.6 

-5 

-3 

-2.2 

-1.3 

Occupational 

CrfSer 

1116 

4.4 

2.5 

3.3 

-21 

-12 

-1.9 

-1.1 

Group 

ElMeRe 

3018 

4.6 

2.0 

2.9 

-79 

-51 

-2.6 

-1.7 

SkilTe 

2863 

3.6 

2.1 

2.9 

-44 

-21 

-1.5 

-.7 

Suppor 

1678 

4.4 

2.7 

3.2 

-29 

-20 

-1.7 

-1.2 

Grade 

3 

279 

8.6 

1.1 

3.9 

-21 

-13 

-7.5 

-4.7 

4 

4920 

5.0 

2.9 

3.2 

-103 

-89 

-2.1 

-1.8 

5 

3454 

2.2 

1.5 

2.7 

-24 

19 

-.7 

.6 

Term  of 

4 

5132 

4.7 

2.3 

3.2 

-126 

-80 

-2.5 

-1.6 

Enlistment 

5 

251 

3.2 

.8 

2.8 

-6 

-1 

-2.4 

-.4 

6 

3399 

3.6 

2.3 

2.9 

-45 

-27 

-1.3 

-.8 

Term  Year 

1 

3459 

4.1 

1.8 

3.4 

-80 

-26 

-2.3 

-.8 

2 

2822 

4.6 

2.5 

2.8 

-60 

-52 

-2.1 

-1.8 

3 

1941 

4.4 

2.6 

2.8 

-34 

-31 

-1.8 

-1.6 

4 

366 

2.5 

2.2 

3.0 

-1 

2 

-.3 

.5 

5 

197 

3.6 

2.5 

3.0 

-2 

-1 

-1.0 

-.5 

Year  of 

3 

1662 

4.3 

2.0 

3.5 

-39 

-14 

-2.3 

-.8 

of  Service 

4 

2103 

4.8 

2.4 

3.1 

-51 

-36 

-2.4 

-1.7 

5 

2282 

4.4 

2.5 

2.9 

-44 

-34 

-1.9 

-1.5 

6 

1337 

3.5 

2.3 

2.8 

-16 

-10 

-1.2 

-.7 

7 

692 

3.8 

2.0 

2.7 

-12 

-7 

-1.7 

-1.0 

8 

393 

3.1 

1.8 

2.8 

-5 

-1 

-1.3 

-.3 

9 

179 

3.9 

2.2 

2.8 

-3 

-2 

-1.7 

-1.1 

Table  4.11 


SECOND-TERM  ETS,  FY  82 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at  -  - 


Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Bonus 

601 

14.5 

19.6 

57 

31 

9.5 

5.2 

386 

22.3 

23.1 

8 

11 

2.1 

2.8 

Occupational 

CrfSer 

133 

19.5 

21.8 

21.1 

3 

2 

2.3 

1.5 

Group 

ElMeRe 

276 

15.9 

22.8 

21.0 

19 

14 

6.9 

5.1 

SkilTe 

391 

18.4 

25.6 

22.8 

28 

17 

7.2 

4.3 

Suppor 

237 

15.2 

20.7 

19.0 

13 

9 

5.5 

3.8 

Grade 

4 

252 

16.7 

30.6 

21.4 

35 

12 

13.9 

4.8 

5 

769 

17.4 

21.2 

21.3 

29 

30 

3.8 

3.9 

Term  of 

4 

783 

17.5 

24.3 

20.4 

53 

23 

6.8 

2.9 

Enlistment 

6 

233 

15.0 

20.2 

22.7 

12 

18 

5.2 

7.7 

Term  Year 

4 

783 

17.5 

24.3 

20.4 

53 

23 

6.8 

2.9 

6 

233 

15.0 

20.2 

22.7 

12 

18 

5.2 

7.7 

Year  of 

6 

418 

17.9 

26.6 

22.2 

36 

18 

8.6 

4.3 

Service 

7 

220 

13.6 

25.5 

18.2 

26 

10 

11.8 

4.5 

8 

295 

21.0 

20.3 

22.0 

-2 

3 

-.7 

1.0 

Table  4,12 

SECOND-TERM  ETS,  FY  84 


Airmen  Loss  Rate  (%)  PF.  (%)  PEPNR  (%) 

Stratification  at 


Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Bonus 

0.0 

840 

16.3 

23.6 

13.1 

61 

-27 

7.3 

-3.2 

1.0 

147 

18  A 

71.1 

13.6 

4 

-7 

2.7 

-4.8 

Occupational 

CrfSer 

144 

15.3 

22.9 

11.8 

11 

-5 

7.6 

-3.5 

Group 

ElMeRe 

218 

15.1 

22.0 

12.4 

15 

-6 

6.9 

-2.8 

SkilTe 

343 

25.7 

28.0 

16.6 

8 

-31 

2.3 

-9.0 

Suppor 

331 

10.3 

19.6 

11.5 

31 

4 

9.4 

1.2 

Grade 

4 

193 

23.8 

31.1 

14.5 

14 

-18 

7.3 

-9.3 

5 

795 

15.3 

21.9 

13.2 

52 

-17 

6.5 

-2.1 

Year  of 

6 

339 

22.4 

27.7 

13.9 

18 

-29 

5.3 

-8.6 

Service 

7 

314 

14.6 

23.9 

12.1 

29 

-8 

9.2 

-2.5 

8 

228 

14.5 

20.2 

14.5 

13 

0 

5.7 

0.0 

Stratification 

Factor 

Level 

Airmen 

Loss  Rate  (%) 

PE  (%) 

PEPNR  (%) 

at 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Occupational 

CrfScr 

79 

6.3 

13.9 

15.2 

6 

7 

7.6 

8.9 

Group 

ElMeRe 

207 

7.7 

15.5 

11.1 

16 

7 

7.7 

3.4 

SkilTe 

331 

10.3 

19.0 

16.6 

29 

21 

8.8 

6.3 

Suppor 

188 

8.0 

14.4 

11.2 

12 

6 

6.4 

3.2 

Term  of 

4 

631 

8.2 

16.2 

13.3 

50 

32 

7.9 

5.1 

Enlistment 

6 

147 

10.2 

18.4 

15.0 

12 

7 

8.2 

4.8 

Term  Year 

5 

397 

8.6 

19.6 

15.6 

44 

28 

11.1 

7.1 

6 

167 

7.8 

10.8 

10.2 

5 

4 

3.0 

2.4 

7 

184 

10.9 

16.8 

14.1 

11 

6 

6.0 

3.3 

Year  of 

7 

266 

10.2 

19.5 

16.2 

25 

16 

9.4 

6.0 

Service 

8 

202 

7.9 

13.9 

12.4 

12 

9 

5.9 

4.5 

9 

210 

10.5 

17.1 

14.8 

14 

9 

6.7 

4.3 

Table  4. 14 

SECOND-TERM  EXTENSION,  FY  84 

Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at  - 

Factor  Level  Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


Occupational 

CrfScr 

68 

11.8 

Group 

ElMeRe 

114 

7.0 

SkilTe 

163 

8.6 

Suppor 

189 

10.6 

Term  Year 

5 

281 

10.0 

6 

107 

9.3 

Year  of 

7 

111 

11.7 

Service 

8 

144 

10.4 

9 

143 

11.9 
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predicted  losses  for  EFMS  are  probably  because  many  such  airmen  are  calculated  to  end 
the  fiscal  year  subject  to  a  retirement  model,  which  predicts  much  higher  losses  than  seen 
in  the  career  attrition  model.  The  higher  ALPS  losses  probably  reflect  the  same 
reasoning  in  some  way,  although  the  phenomenon  is  much  weaker  in  FY  84  than  in  FY 
82.  The  anomaly  suggests  some  problem  exists  in  the  YAR  file  with  the  coding  of  the 
year  of  service  or  with  the  evaluation  code’s  decision  of  when  it  is  appropriate  to  use  the 
retirement  model.  The  ALPS  model  makes  the  same  mistake,  at  least  in  FY  82, 
suggesting  some  problem  with  the  YAR  file;  but  ALPS  does  not  make  as  big  a  mistake  as 
EFMS,  implicating  the  evaluation  code. 

About  350  fewer  airmen  are  in  their  18th  year  of  service  than  in  their  19th  year  of 
service  in  both  fiscal  years.  Looking  ahead  to  Tables  4.21  and  4.22,  which  describe 
retirement-eligible  airmen,  there  are  some  300  airmen  in  their  18th  year  of  service, 
compared  with  more  than  700  in  their  19th.  In  FY  82,  both  models  undere slimale  losses 
for  airmen  in  their  18th  year  of  service.  This  may  be  further  evidence  of  some 
misassignment  of  airmen  in  their  18th  year  of  service  to  career  or  retirement  models. 

Career  ETS 

Tables  4. 1 7  and  4. 18  compare  EFMS  and  ALPS  for  the  career  ETS  model  for  FY 
82  and  FY  84  respectively. 

Career  Extension 

Tables  4.19  and  4.20  compare  EFMS  and  ALPS  for  the  career  extension  model  for 
FY  82  and  FY  84  respectively. 

Retirement 

Tables  4.21  and  4.22  compare  EFMS  and  ALPS  for  the  retirement  model  for  FY 
82  and  FY  84  respectively. 

COMPARISON  BY  OCCUPATIONAL  GROUPS 

This  section  compares  end-strength  predictions  of  the  EFMS  middle-term 
disaggregate  loss  models  with  ALPS  predictions  by  AFSC.  Each  of  the  approximately 
400  AFSC  groups  found  at  the  beginning  of  each  fiscal  year  was  used  with  the  EFMS 
models  to  estimate  how  many  airmen  would  be  present  at  the  end,  and  then  these 


Table  4.15 

CAREER  ATTRITION,  FY  82 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at 

Factor  Level  Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


Occupational  CrfSer 
Grouo  ElMcRe 


Suppor 

2239 

Grade 

4 

239 

5 

3913 

6 

3127 

7 

1048 

Term  of 

3 

123 

Enlistment 

4 

5751 

5 

866 

6 

1677 

Term  Year 

1 

2980 

2 

2643 

3 

2174 

4 

399 

5 

246 

Year  of 

7 

417 

Service 

8 

633 

Table  4.17 


CAREER  ETS,  FY  82 


Stratification 

Factor 

Level 

Airmen 

at 

Risk 

Loss  Rate  (%) 

PE  (%) 

PEPNR  (%) 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Bonus 

0.0 

1526 

2.9 

3.8 

5.7 

13 

42 

.9 

2.8 

1.0 

124 

6.5 

4.8 

6.5 

-2 

0 

-1.6 

0.0 

Occupational 

CrfSer 

234 

3.0 

4.3 

6.8 

3 

9 

1.3 

3.8 

Group 

ElMeRe 

489 

2.5 

2.9 

5.3 

2 

14 

.4 

2.9 

SkilTe 

475 

3.6 

5.1 

6.5 

7 

14 

1.5 

2.9 

Suppor 

461 

4.1 

3.3 

5.2 

-4 

5 

-.9 

1.1 

Grade 

5 

721 

5.0 

6.0 

6.9 

7 

14 

1.0 

1.9 

6 

714 

2.1 

2.2 

4.5 

1 

17 

.1 

2.4 

7 

198 

0.0 

2.5 

6.1 

5 

12 

2.5 

6.1 

Term  of 

4 

1414 

3.1 

3.7 

5.7 

9 

36 

.6 

2.5 

Enlistment 

6 

172 

4.1 

3.5 

5.2 

-1 

2 

-.6 

1.2 

Term  Year 

4 

1414 

3.1 

3.7 

5.7 

9 

36 

.6 

2.5 

6 

172 

4.1 

3.5 

5.2 

-1 

2 

-.6 

1.2 

Year  of 

10 

412 

5.3 

5.3 

6.6 

0 

5 

0.0 

1.2 

Service 

11 

182 

6.0 

3.8 

6.0 

-4 

0 

-2.2 

0.0 

12 

— 

13 

156 

1.9 

1.9 

4.5 

0 

4 

0.0 

2.6 

14 

265 

.4 

1.1 

3.0 

2 

7 

.8 

2.6 

15 

237 

.8 

.8 

3.4 

0 

6 

0.0 

2.5 

16 

— 

NOTE:  There  were  too  few  airmen  at  12  and  16  years  of  service  to  make  reliable  estimates. 


estimates  were  compared  with  those  of  ALPS.  The  findings  are  summarized  in  Table 
4.23. 

The  EFMS  model  appears  to  do  belter  than  ALPS  in  FY  82,  but  the  differences  are 
not  as  great  in  FY  84,  probably  because  economic  factors  did  not  change  so  much  in  FY 
84;  and  data  from  FY  82  were  used  to  estimate  the  EFMS  model,  but  data  from  FY  84 
were  not. 


Table  4.23  treats  each  AFSC  as  equally  important,  but  some  AFSCs  have 
thousands  of  airmen,  and  others  have  only  a  few.  Table  4.24  compares  the  two  models  in 
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Table  4.18 

CAREER  ETS.FY84 


Airmen 

Loss  Rate  (%) 

PE  (%) 

PEPNR  (%) 

Stratification 

at 

Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Occupational 

CrfSer 

207 

2.4 

2.4 

2.9 

0 

1 

0.0 

.5 

Group 

ElMeRe 

482 

2.3 

2.7 

3.1 

2 

4 

.4 

.8 

SkilTe 

435 

4.4 

3.4 

3.0 

-4 

-6 

-.9 

-1.4 

Suppor 

399 

2.3 

2.5 

2.8 

1 

2 

.3 

.5 

Grade 

5 

544 

4.0 

4.6 

3.9 

3 

-1 

.6 

-.2 

6 

727 

1.5 

1.7 

2.5 

1 

7 

.1 

1.0 

7 

224 

.9 

1.8 

2.2 

2 

3 

.9 

1.3 

Term  of 

4 

1291 

2.7 

2.8 

3.0 

1 

4 

.1 

.3 

Enlistment 

6 

172 

5.2 

2.3 

2.9 

-5 

-4 

-2.9 

-2.3 

Term  Year 

4 

1291 

2.7 

2.8 

3.0 

1 

4 

.1 

.3 

6 

172 

5.2 

2.3 

2.9 

-5 

-4 

-2.9 

-2.3 

Year  of 

10 

207 

5.3 

4.8 

3.9 

-1 

-3 

-.5 

-1.4 

Service 

11 

182 

3.8 

3.3 

3.3 

-1 

-1 

-.5 

-.5 

12 

132 

4.5 

3.0 

3.0 

-2 

-2 

-1.5 

-1.5 

13 

156 

2.6 

1  3 

2.6 

-2 

0 

-1.3 

0.0 

14 

343 

1.5 

1.2 

2.9 

-1 

5 

-.3 

1.5 

15 

280 

1.4 

.7 

2.5 

-2 

3 

-.7 

1.1 

16 

100 

2.0 

1.0 

3.0 

-1 

1 

-1.0 

1.0 

Table  4.19 

CAREER  EXTENSION,  FY  82 

Airmen 

Loss  Rate  (%) 

PE  (%) 

PEPNR  (%) 

Stratification 

at 

Factor 

Level 

Risk 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Occupational 

CrfSer 

62 

1.6 

9.7 

3.2 

5 

1 

8.1 

1.6 

Group 

ElMeRe 

169 

2.4 

4.1 

5.3 

3 

5 

1.8 

3.0 

SkilTe 

229 

3.5 

6.1 

5.2 

6 

4 

2.6 

1.7 

Suppor 

175 

1.7 

5.1 

4.0 

6 

4 

3.4 

2.3 

Grade 

5 

229 

2.6 

5.2 

6.1 

6 

8 

2.6 

3.5 

6 

305 

3.0 

5.6 

3.6 

8 

2 

2.6 

.7 

Term  Year 

5 

351 

1.7 

5.7 

4.6 

14 

10 

4.0 

2.8 

6 

165 

3.6 

4.8 

4.2 

2 

1 

1.2 

.6 

Year  of 

11 

156 

3.2 

5.1 

5.1 

3 

3 

1.9 

1.9 

Service 

12 

117 

1.7 

4.3 

4.3 

3 

3 

2.6 

2.6 
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Table  4.20 


CAREER  EXTENSION,  FY  84 


Stratification 

Factor 

Level 

Airmen 

at 

Risk 

Loss  Rate  {%) 

PE  (%) 

PEPNR  (%) 

Actual 

EFMS 

ALPS 

EFMS 

ALPS 

EFMS 

ALPS 

Occupational 

CrfSer 

59 

5.1 

8.5 

1.7 

2 

-2 

3.4 

-3.4 

Group 

ElMeRe 

135 

2.2 

3.7 

2.2 

2 

0 

1.5 

0.0 

SkilTe 

122 

5.7 

6.6 

2.5 

1 

-4 

.8 

-3.3 

Suppor 

132 

.8 

4.5 

2.3 

5 

2 

3.8 

1.5 

Grade 

5 

121 

6.6 

5.0 

3.3 

-2 

-4 

-1.7 

-3.3 

6 

248 

1.6 

5.6 

1.6 

10 

0 

4.0 

0.0 

Term  Year 

5 

237 

1.7 

5.5 

2.1 

9 

1 

3.8 

.4 

6 

103 

4.9 

3.9 

1.9 

-1 

-3 

-1.0 

-2.9 

a  different  way  and  shows  how  the  two  models  compare  when  differences  in  AFSC  sizes 
arc  taken  into  account. 

For  each  AFSC,  the  absolute  value  of  each  model’s  prediction  error  was 
calculated  to  see  whether  the  EFMS  prediction  was  better  (absolute  value  of  error  closer 
to  zero),  the  same,  or  worse  than  ALPS.  Table  4.24,  reading  down  the  first  column  in 
each  fiscal  year  panel  under  AFSC,  shows  the  percent  of  all  AFSCs  in  which  EFMS  did 
better,  the  same,  or  worse  than  ALPS  respectively.  In  each  fiscal  year,  ALPS  performed 
strictly  better  than  EFMS  on  less  than  a  quarter  of  the  AFSCs. 

In  the  second  column,  the  comparison  was  weighted  by  the  number  of  airmen  in 
the  associated  AFSC.  A  number  in  an  Airmen  column  may  be  thought  of  as  the 
probability  that  an  "average"  airman  will  belong  to  an  AFSC  where  EFMS  does  better, 
the  same,  or  worse  than  ALPS,  respectively.  As  can  be  seen,  in  FY  82  only  10  percent  of 
the  force  belonged  to  AFSCs  in  which  ALPS  did  better.  The  percent  where  EFMS  is  the 
same  as  ALPS  is  considerably  smaller  in  the  weighted  column  than  in  the  unweighted 
column  largely  because  it  is  easier  for  the  two  models  to  agree  on  the  many  small  AFSCs 
containing  only  one  or  two  sampled  airmen. 

It  is  hard  to  compare  the  two  models  on  small  AFSCs.  These  smaller  AFSCs  were 
eliminated  from  subsequent  analysis  by  sorting  them  from  largest  to  smallest;  and  the 
cumulative  number  of  ainnen  obtained  was  calculated  from  larger  to  smaller.  When  the 
accumulated  sum  was  90  percent  of  the  sample  of  airmen,  all  smaller  ones  were  dropped. 
So  the  following  observations  apply  to  approximately  90  percent  of  the  Air  Force. 
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Table  4.21 

RETIREMENT,  FY  82 


Airmen  Loss  Rate  (%)  PE  (%)  PEPNR  (%) 

Stratification  at  -  - 

Factor  Level  Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


. . . 


Tabic  4.22 

RETIREMENT,  FY  84 
Loss  Rate  (%) 


Airmen  Loss  Rate  (%)  PE(%)  PEPNR  (%) 

at  -  - 

Risk  Actual  EFMS  ALPS  EFMS  ALPS  EFMS  ALPS 


PEPNR  (%) 


Stratification 

Factor 


Statistic 


EFMS  ALPS  EFMS  ALPS 


Average  Absolute  Error  3.0  4.2  2.1  2.0 

Root  Mean  Square  Error  8.6  12.9  4.9  4.3 


Table  4.24 

PEkCENT  OF  AFSCs  AND  AIRMEN  WHERE  EFMS  DOES  BETTER 


AFSCs  where: 

FY  82 

FY  84 

AFSC 

Airmen 

AFSC 

Airmen 

EFMS  better  than  ALPS 

32 

75 

27 

43 

EFMS  same  as  ALPS 

41 

10 

45 

14 

EFMS  worse  than  ALPS 

27 

19 

28 

43 

To  standardize  predictions  across  AFSCs  the  prediction  error  (predicted  end 
strength  minus  actual)  of  each  model  was  divided  by  the  number  of  airmen  at  risk. 
Figures  5  and  6  plot  these  estimates,  restricted  to  the  "larger"  AFSCs,  for  each  model. 
For  clarity,  the  sample  was  sorted  by  ascending  values  of  the  ALPS  error. 

In  Figs.  5  or  6,  a  perfect  model  would  have  zero  error,  and  its  graph  would 
coincide  with  the  axis  line  running  across  the  middle  of  each  display.  The  closer  the 
graph  of  a  real  model  lies  to  this  axis,  the  better  it  is.  It  is  clear  that  in  FY  82,  the  EFMS 
model  performance  is  superior  to  that  of  ALPS.  In  FY  84,  the  EFMS  model  appears  to 
do  better  over  those  AFSCs  where  the  ALPS  model  underpredicts  losses,  but  it  tends  to 
do  worse  than  the  EFMS  model  over  those  AFSCs  where  the  ALPS  model  overpredicts 
losses. 

Finally,  each  of  the  samples  of  "large"  AFSCs  was  sorted  by  size,  largest  to 
smallest  and  then  the  cumulative  absolute  error  calculated  for  each  model.  This 
cumulative  sum  was  plotted  against  the  fraction  of  the  force  covered  by  the  sum.  These 
plots  are  shown  in  Figs.  7  and  8.  A  perfect  model  would  have  absolute  errors  equal  to 
zero  in  each  AFSC,  and  so  its  curve  would  lie  flat  along  the  x-axis.  The  closer  the  graph 
of  a  real  model  lies  to  this  axis,  the  better  it  is.  As  can  be  seen,  in  FY  82  the  two  curves 
diverge  sharply,  indicating  a  superior  performance  of  the  EFMS  model  across  many 
AFSCs.  In  FY  84,  the  EFMS  shows  a  slightly  superior  performance  for  the  larger 
AFSCs. 


EFMS 
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